Jump to content
  • Announcements

    • admin

      PBS Forum Has Closed   06/12/17

      The PBS Works Support Forum is no longer active.  For PBS community-oriented questions and support, please join the discussion at http://community.pbspro.org.  Any new security advisories related to commercially-licensed products will be posted in the PBS User Area (https://secure.altair.com/UserArea/). 
Sign in to follow this  
jburri

Distributing executables to chunks

Recommended Posts

I have a large program that is broken up into a number of executable/processes that communicate via MPI. The processes use varying amounts of computer resources and I would like to group them so that processes that use a large amount of memory and CPU cycles are on their own chunk and less intensive processes are grouped together.

I understand how to define chunks with different amounts of resources, but what I don’t understand is how to direct processes to certain chunks. For example, if I have the following three processes:

proc1 uses 500mb of mem

proc2 uses 10 gb of mem

proc3 uses 1gb of mem

I would like to define one chunk with 10gb of memory where exe2 is run and one with 1.5gb of memory where exe1 and exe3 are run. How do I accomplish this?

I have been defining the chunks, but the processes get put into the wrong chunks and the large processes are exceeding the memory limits on the small chunks and my jobs die.

Any assistance you can provide would be greatly appreciated.

By the way, I am using PBS Pro 9.1.

-Jeremy

Share this post


Link to post
Share on other sites

Can you tell me which MPI you are using? It should be possible using your MPI options (depending upon what you are using) to pass an executable to each rank either via an "appfile" or some other option. Knowing what MPI you are using would allow for more specific advice. I think basically the answer is that once PBS assigns you the resources its is really up to you to use those resources the way you need to in the job script. There may be some other ways but the appfile seems easiest.

Share this post


Link to post
Share on other sites

Hi Adiaz,

I am using MPI/PRO 2.2.

I think I understand what to do. I use the PBS_NODEFILE and build my "appfile" to make sure that my processes land in the right location.

Thanks for the help.

-Jeremy

Share this post


Link to post
Share on other sites

Hi Adam,

I have been able to get things working correctly, but a different sort of problem has surfaced. The memory tracked and reported by PBS is changing depending on whether or not I have all of my chunks on one node or spread of multiple nodes.

To illustrate what I an see, consider the following example:

proc1 uses x mb of mem

proc2 uses y gb of mem

proc3 uses z gb of mem

If I define a separate chunk for each processes but add "-l place=pack", PBS will report that I used 10 gb of memory. If instead I add "-l place=scatter", PBS will report that I used 7.5 gb of memory. The cpu time is also substantially less for the "-l place=scatter" case. Does this make any sense to you?

Of course, this could be an issue with my code, but this seems to be more of a PBS accounting issue.

Is this a known deficiency with PBS? I know older versions of PBS Pro did not account for parallel jobs very well.

Thanks again for any insight you can provide.

-Jeremy

Share this post


Link to post
Share on other sites

I think this has to do with your MPI. If you check out the admin guide (section 11.10 in the 9.1 AG) there is a section on integration with MPI. PBS needs to be able to use what is called "pbs_attach" to allow it to monitor, cleanup and control all the processes in a PBS job. Some MPI vendors do this for you and provide a configure option when you build it (like OpenMPI). For others we can use a simple scripted integration by redirecting the mpirun command to a pbs version of that command. It essentially takes over the np and machine file args (passing all other options though) and make use of the pbs_attach command. You basically have two options:

1. take an exising pbs_mpirun and edit it for your use (see 11.10.1.3)

2. make use of the pbsun_wrap - while ultimately more flexible this require more work.

You simply need to setup the mpi integration correctly and the accounting should work correctly.

Share this post


Link to post
Share on other sites

Please sign in to comment

You will be able to leave a comment after signing in



Sign In Now
Sign in to follow this  

×