Jump to content
  • Announcements

    • admin

      PBS Forum Has Closed   06/12/17

      The PBS Works Support Forum is no longer active.  For PBS community-oriented questions and support, please join the discussion at http://community.pbspro.org.  Any new security advisories related to commercially-licensed products will be posted in the PBS User Area (https://secure.altair.com/UserArea/). 
Sign in to follow this  
jmesseng

How-to for PBS Error: Floating License unavailable

Recommended Posts

PBS Pro license issue: Not Running: PBS Error: Floating License unavailable”

 

09-May-2014

 

System:

  • PBS Pro version 12.0 installed on SLES 11 SP2
  • LMX version 12.0 license manager
  • Network license schema using a three server HAL setup

              

Symptom:

            qstat –sw  reports   “Not Running: PBS Error: Floating License unavailable

  • One or more jobs in Q state affected
  • Potentially your LMX license server is down and needs to be restarted
  • Potentially an update was done to LMX licensing such as new license file and despite normal appearances where other Altair LMX based applications are able to check-out licenses (look at LMX logs on the license server and run lmx license stat)  PBS Pro is reporting this problem
  • Existing jobs in the PBS queue may be running unaffected
  • Some new jobs submitted to the PBS queue may transition to R state

 

Notes:

  • Attempting to qrun an affected job fails because PBS cannot allocate needed licenses
  • Verify PBS license location string is correct
    • If using a HAL schema, the host order in the string has to match the HAL server order in the LMX server configuration file, altair-serv.cfg
  • Sending the PBS scheduler a HUP will not work
  • You do not want to attempt a qterm and warm start of the PBS server in the given license circumstance

 

 

How-to resolve:

Re-insert the pbs_license_info string using qmgr, this causing PBS to reinitialize the licensing internally

 

  • Use    qmgr -c "p s" | grep pbs_license_info   to display the license string and a good time to confirm it is correct, see Notes above

  • Using the license string captured above or a corrected one, reinitialize the PBS license connection:

           qmgr –c “set server pbs_license_file_location = 6200@huey:6200@louie:6200@dewey

 

 

Example of condition:

qstat -ws

vulcan:
                                                                                                   Req'd  Req'd   Elap
Job ID                         Username        Queue           Jobname         SessID   NDS  TSK   Memory Time  S Time
------------------------------ --------------- --------------- --------------- -------- ---- ----- ------ ----- - -----
106976.vulcan                krockon         gp              MRB_FC              3221    2     4  126gb   --  R 76:20:11
   Job run at Tue May 06 at 05:56 on (vulcan[22]:ncpus=2:mem=66060288kb)+(vulcan[23]:ncpus=2:mem=66060288kb)
107077.vulcan                choosew         cfd_gp          Cfg6_CP_2         279272   10    60  300gb   --  R 65:52:12
   Job run at Tue May 06 at 16:24 on (vulcan[62]:ncpus=6:mem=31457280kb)+(vulcan[63]:ncpus=6:mem=31457280kb)+(...
107282.vulcan                hammero         gp              HX22E_full_f      379874    1     4  254gb   --  R 02:59:10
   Job run at Fri May 09 at 07:17 on (vulcan[30]:mem=67108864kb:ncpus=4+vulcan[31]:mem=67092480kb+vulcan[28]...
107293.vulcan                nuktynm         shortgp         nxn_43n_bm           --     1     4   63gb 01:00 Q  --
  
Not Running: PBS Error: Floating License unavailable
107294.vulcan                nuktynm         shortgp         nxn_43m_bm           --     1     6   63gb 01:00 Q  --
  
Not Running: PBS Error: Floating License unavailable
107299.vulcan                aaelidk         gp              SOF5_Heb_SYS12       --     1     4   63gb   --  Q  --
  
Not Running: PBS Error: Floating License unavailable
107301.vulcan                saracki         gp              MCDV-Down-Af      158128    1     1  254gb   --  R 00:04:14
   Job run at Fri May 09 at 10:12 on (vulcan[26]:mem=67108864kb:ncpus=1+vulcan[27]:mem=67092480kb+vulcan[24]...

 

Share this post


Link to post
Share on other sites

Please sign in to comment

You will be able to leave a comment after signing in



Sign In Now
Sign in to follow this  

×