Jump to content
  • Announcements

    • admin

      PBS Forum Has Closed   06/12/17

      The PBS Works Support Forum is no longer active.  For PBS community-oriented questions and support, please join the discussion at http://community.pbspro.org.  Any new security advisories related to commercially-licensed products will be posted in the PBS User Area (https://secure.altair.com/UserArea/). 
mike

disk space for jobs

Recommended Posts

Hello,


 


Can you tell me if there is a "qmgr set queue" command that I can set to allow users to specify, in their jobs, how much local disk space is used per job? We have nodes where there is only a limited amount of available /tmp space, and we would like to have qmgr direct any job with a higher required disk space to another set of nodes, automatically. I'm fairly new to PBS and haven't been able to locate any information on this topic.


 


Thanks for your time,


Mike


Share this post


Link to post
Share on other sites

Hi Mike, 


 


There is a way to customize PBS to schedule jobs based on available disk space. You could schedule based on dynamic host-level resources or static host-level resources. There are some advantages and disadvantages for the different ways to configure. Personally, I lean towards the static host-level resource option.


 


When a job requests a dynamic resource, PBS checks to see how much of the resource is available, but cannot know how much will be used by another job while this job executes. This can lead to a resource shortage. For example, there is 20GB of scratch on a disk, no jobs are running, and a job requests 15GB. This job writes 5GB during the first part of its execution, then another job requests 10GB. The second job is started by PBS, because there is 15GB available. Now there is a shortage of scratch space.


 

You can avoid this problem by configuring a static consumable resource to represent scratch space. Set it to the amount of available scratch space. 

 

I will provide you examples for both configurations while referencing the PBS Professional v12.2 Administrator Guide. 

 


5.14.5.1.i Example of Configuring Dynamic Host-level Resource, but shorten up the instructions. 


  1. Write a script, for example scratch.sh, that returns the available amount of the resource via stdout. The script must return the value in a single line, ending with a newline. Place the script on each host where it will be used. For example, it could be placed in /usr/local/bin/scratch.sh.

#!/bin/sh
# Small script which determines available 'scratch' space which
# MOM executes as a dynamic resource

# If there are no arguements passes with the command, then
# assume that /s1 is the location of the 'local scratch'
[ $# -gt 0 ] && PBS_WD=$1 || PBS_WD=/s1

case `uname -s` in
  AIX)     free=`df -k $PBS_WD | awk 'NR==2 {print $3}'` ;;
  Linux)   free=`df -kP $PBS_WD | awk 'NR==2 {print $4}'` ;;
  IRIX*)   free=`df -k $PBS_WD | awk 'NR==2 {print $5}'` ;;
  SunOS)   free=`df -k $PBS_WD | awk 'NR==2 {print $4}'` ;;
  OSF1)    free=`df -k $PBS_WD | awk 'NR==2 {print $4}'` ;;
  HP-UX)   free=`bdf $PBS_WD | awk 'NR==2 {print $4}'` ;;
  *)              free=0 ;;
esac

# Return the units in megabytes
free=`expr $free / 1024`
echo "${free}mb"

 

On each compute node (running pbs_mom)


  1. Configure each pbs_mom to use the script by adding the resource and the path to the script in PBS_HOME/mom_priv/config:
    1. scratch !/usr/local/bin/scratch.sh

  2. Reinitialize the MoMs by sending a HUP signal
    1. kill -HUP `cat PBS_HOME/mom_priv/mom.lock`

On the PBS Server/Scheduler


  1. Define the resource, for example scratch, in the server resource definition file PBS_HOME/server_priv/resourcedef:
    1. scratch type=size flag=h

  2. Add the new resource (scratch) to the resources: line of the scheduler configuration file PBS_HOME/sched_priv/sched_config
    1. resources: “ncpus, mem, arch, [...], scratch”

  3. Add the new resource (scratch) to the mom_resources: line of the scheduler configuration file PBS_HOME/sched_priv/sched_config
    1. mom_resources: “scratch”

  4. Restart the server and scheduler.
    1. qterm -t quick -s ; /etc/init.d.pbs start

To request this resource, the resource request would include


 


qsub -l select=1:ncpus=N:scratch=1gb job_script.sh


 


 


5.14.5.2.i Example of Configuring Static Host-level Resource


In this example, we configure a consumable host-level resource to track disk space. 

 

On the PBS Server/Scheduler

  1. Define the resource, for example scratch, in the server resource definition file PBS_HOME/server_priv/resourcedef:
    1. scratch type=size flag=nh


  2. Add the new resource (scratch) to the resources: line of the scheduler configuration file PBS_HOME/sched_priv/sched_config


    1. resources: “ncpus, mem, arch, [...], scratch”




  3. Restart the server and scheduler.


    1. qterm -t quick -s ; /etc/init.d.pbs start




  4. Use the qmgr command to set the value of the resource on each host:

    1. PBS_EXEC/bin/qmgr -c "set node <nodename> resources_available.scratch=<value of disk space with units>"
    2. qmgr -c "set node node001 resources_available.scratch=20gb"


To request this resource, the resource request would include

 

qsub -l select=1:ncpus=N:scratch=1gb job_script.sh

 

BTW, if you want to change TMPDIR or have jobs execute in a different partition, you may want to review how to configure 11.13.1 The Job’s Staging and Execution Directories and 11.14.1 Creation of TMPDIR in the PBS Professional v12.2 Administrators Guide.


 


Extra credit .. If you want to record how much disk space the job 'actually' used - at the end of execution - you would will want to use a execjob_epilogue hook. 


Share this post


Link to post
Share on other sites

Outstanding. Thanks to you, sir. I appreciate your time.


 


Will this ":scratch=1gb" simply be ignored if the user doesn't need to define a job size? Usually, jobs are run using a shared storage array. This additional change will be for those users who wish to specifically run a job on the local disk of any particular node.


 


---


Last question, a user was trying to use this command, within their job, to allocate disk space:


 


#PBS -l file=750gb


 


I looked into this pbs qsub directive, but thought that this directive would only limit the file size to 750gb, hence my original question. I am running PBSPro 12.2.1.140562, on linux os. Am I interpreting this directive correctly?


 


 


Thanks again,


Mike


Share this post


Link to post
Share on other sites

WRT ":scratch=1gb", the user will need to specify this at submission if they require it. Otherwise, the ":scratch=XX" is optional. Thinking about it, I should have called it tmp or local_tmp instead of scratch. 


 


WRT "-l file=750gb", you are correct that specifying the file attribute will limit the file size per-process. On a linux system, this attribute will set RLIMIT_FSIZE with the kernel. If the job's process creates a file that exceeds this limit, the kernel will kill the job and it will exit PBS. 


Share this post


Link to post
Share on other sites

Please sign in to comment

You will be able to leave a comment after signing in



Sign In Now

×