Jump to content
  • Announcements

    • admin

      PBS Forum Has Closed   06/12/17

      The PBS Works Support Forum is no longer active.  For PBS community-oriented questions and support, please join the discussion at http://community.pbspro.org.  Any new security advisories related to commercially-licensed products will be posted in the PBS User Area (https://secure.altair.com/UserArea/). 
Davide Pastorino

Hook on execjob_epilogue and epilogue script execution

Recommended Posts

Hi all,

we're trying to use an hook to requeue a job in some specific conditions* but we also need to execute the epilogue script if the requeue is not deemed necessary.

Is there a built in function do execute the epilogue from a hook?

If not, is something like

subprocess.Popen(["rm","-r","some.file"])

acceptable within a hook?

Or is there any other option?

 

*(e.g. job fail due to unavailable license: we're using dynamic resources for scheduling but since the licenses are not exclusive to the cluster there's always the possibility of a race condition with the workstation users...)

Share this post


Link to post
Share on other sites

Hello,

I think that is indeed the best way to go about it, and it's definitely acceptable in a hook (make sure you have a sensible hook alarm defined in qmgr, in case the script hangs for any reason).  One big caveat though is that Python goes in and sets up a lot of its own signal handlers, and a Linux shell script is going to be expecting those handlers to be different than they are (this can be a problem if, for example, your epilogue or prologue contains any pipes).  Fortunately, with the Popen approach we can work around that following a trick like this: http://www.chiark.greenend.org.uk/~cjwatson/blog/python-sigpipe.html

 

Quote

import signal
import subprocess

def subprocess_setup():
    # Python installs a SIGPIPE handler by default. This is usually not what
    # non-Python subprocesses expect.
    signal.signal(signal.SIGPIPE, signal.SIG_DFL)

subprocess.Popen(command, preexec_fn=subprocess_setup)

 

Hope this helps!

 -- Matt

Share this post


Link to post
Share on other sites

A quick update: since pbs_python has integrated access to a few modules and it's v2.5 we went with os instead of subprocess.

First we had to modify the sys.path (default pbs installation paths):

import sys
if "/opt/pbs/default/lib/python/python2.5" not in sys.path:
        sys.path.append("/opt/pbs/default/lib/python/python2.5")

this gave us access to the string modules which we used to get the appropriate parameters to pass to the epilogue script:

e=pbs.event()
s=pbs.server()
j=e.job
jobid=j.id
username=string.split(j.Job_Owner,"@")[0]

Then we use the os.spawnl function to launch a backgrounded epilogue when a requeue is not needed:

os.spawnl(os.P_NOWAIT,"/path/to/epilogue","epilogue",jobid,username,groupname,jobname)

We'll be testing this for a few days, if anything comes up I'll update the post.

Share this post


Link to post
Share on other sites

Please sign in to comment

You will be able to leave a comment after signing in



Sign In Now

×