Jump to content
  • Announcements

    • admin

      PBS Forum Has Closed   06/12/17

      The PBS Works Support Forum is no longer active.  For PBS community-oriented questions and support, please join the discussion at http://community.pbspro.org.  Any new security advisories related to commercially-licensed products will be posted in the PBS User Area (https://secure.altair.com/UserArea/). 
Jakub

using server_dyn_res

Recommended Posts

I'm using server_dyn_res variables to track if a job should run. This is working fine but I would like to be able to run multiple jobs per scheduler iteration. Does PBS increase the values of the variables using the entries set in each job, or do I have to continue only running one job at a time.

 

Thanks

Jakub

Share this post


Link to post
Share on other sites

Hi Jakub, 

The availability of a dynamic server-level resource is determined by running a script or program specified in the server_dyn_res line of PBS_HOME/sched_priv/sched_config. The value for resources_available.<resource> is updated at each scheduling cycle with the value returned by the script. This script is run at the host where the scheduler runs, once per scheduling cycle. The script must return the value via stdout in a single line ending with a newline.

The scheduler tracks how much of each numeric dynamic server-level custom resource has been assigned to jobs, and will not overcommit these resources.

So, lets go through a scenario. Assuming that there was sufficient resources to satisfy the jobs native resource requests (e.g., ncpus, mem). We will have a server_dyn_res script querying for foo_widget. The server_dyn_res script returned the value of 50. So, resources_available.foo_widget=50 for this scheduling cycle. If each job was asking foo_widget=1, then the scheduler could schedule up to 50 jobs. (1job:1foo_widget).  Now, at the next scheduling cycle, the server_dyn_res script would execute and report back a new number. Does this help?

BTW, the PBS Works User Forum is not very active (well besides SPAM) these days. I recommend that you raise your question on the http://community.pbspro.org forum, which has more folks monitoring.  

Scott

Share this post


Link to post
Share on other sites

Hi Scott

Thanks for your reply. And thanks for the point to a different community forum.

So if I understand you correctly in your scenario, then if instead each job used foo_widget=2 then if you had 50 jobs in the queue 25 would run in the first pass of the scheduler, if the script returned 50. Then another 25 would run the next pass if the script returned 50 again. And if I'm correct each job is only processed once per scheduler cycle so if the job finished really quickly a new one would not start during that pass (ie the usage of foo_widget would not be decreased by 2)?

Just to give you some background what I'm doing: we are using pbs pro do govern approval for our vCloud requests. We queue up all the requests and then approve thouse that we have enough resources. So we have a scripts which obtains cpu and memory and assign to server_dyn_res variables and each approval passes its usage a resource requirements. As such we would like to allow as many approvals as we can without over subscribing. 

 

cheers,

Jakub

Share this post


Link to post
Share on other sites

Hello Jakub, 

25 minutes ago, Jakub said:

if instead each job used foo_widget=2 then if you had 50 jobs in the queue 25 would run in the first pass of the scheduler

True

25 minutes ago, Jakub said:

Then another 25 would run the next pass if the script returned 50 again.

True

26 minutes ago, Jakub said:

And if I'm correct each job is only processed once per scheduler cycle

True

26 minutes ago, Jakub said:

if the job finished really quickly a new one would not start during that pass

True

At each scheduling cycle, the server_dyn_res script will execute and set the value of resources_available.foo_widget. The Scheduler will decrement the resources_available.foo_widget value by each job requesting foo_widget until there are no more available resources_available.foo_widget. So, at the next scheduling cycle, the server_dyn_res script will execute and use the value your script returns. 

37 minutes ago, Jakub said:

Just to give you some background what I'm doing: we are using pbs pro do govern approval for our vCloud requests. We queue up all the requests and then approve thouse that we have enough resources. So we have a scripts which obtains cpu and memory and assign to server_dyn_res variables and each approval passes its usage a resource requirements. As such we would like to allow as many approvals as we can without over subscribing. 

WRT your background, I don't believe I see the entire picture. I can imagine several different approaches for using "special" nodes, which I am going to consider you are calling vCloud resources. 

I am assuming that vCloud resources are compute nodes that are somewhere else. For instance, the vCloud resources are compute resources "off site" (maybe in a cloud). Am I assuming correctly?

If this is true, then I am also assuming there is a single PBS Server/Scheduler managing the local compute nodes and the "off site" compute nodes. Is this is true?

It seems to me that you want to make a decision on a job-by-job

Simple approach, if you only have 50 'units' of vCloud, then set this as a static value on the PBS Server (qmgr -c "s s resources_available.vCloud=50), and when the job requests vCloud=<value>, then the Server will keep track of how many vCloud resources are assigned (resources_assigned.vCloud), and the Scheduler will take this into consideration during the scheduling cycle. Although the "decision" to approve the vCloud resource would need to be done earlier in the job submission (e.g., queuejob hook). The queuejob hook will be responsible for making the decision and setting the vCloud value. In addition, the queuejob hook would modify the job's request and add to it an attribute to make sure it runs on the vCloud resources.

You may want to consider moving a job from a queue to the "special" vCloud" queue. the vCloud resources only run jobs from the vCloud queue. 

If you have a PBS Server/Scheduler running on the vCloud resources, you could consider using peer scheduling. 

Maybe a runjob hook could be used to make the decision on whether to run the job or not.

Share this post


Link to post
Share on other sites

Please sign in to comment

You will be able to leave a comment after signing in



Sign In Now

×