<br><font size=2 face="sans-serif">Hi Ashley,</font>
<br>
<br><font size=2 face="sans-serif">May I introduce you my patch against padb r341 for supporting Slurm combined with openmpi orte manager.</font>
<br><font size=2 face="sans-serif">The Key is we use salloc to get resource from slurm and then use it to run mpirun of openmpi to start jobs.</font>
<br><font size=2 face="sans-serif">This kind of combination is not yet supported by current padb so far. When we start padb with</font>
<br><font size=2 face="sans-serif">rmgr=slurm in a such job environment we have seen only the stack of orted (see below)</font>
<br><font size=2 face="sans-serif">So my patch aims to remedy the situation.</font>
<br>
<br><font size=2 face="sans-serif">Here are what's going on:</font>
<br><font size=2 face="sans-serif">salloc -p tsl -w machu139,machu140,machu141</font>
<br><font size=2 face="sans-serif">[thipa@machu0 padb_open]$ </font>
<br><font size=2 face="sans-serif">salloc: Granted job allocation 8324</font>
<br><font size=2 face="sans-serif">[thipa@machu0 padb_open]$ </font>
<br><font size=2 face="sans-serif">[thipa@machu0 padb_open]$ srun -n1 mpirun -n 9 pp_sndrcv_spbl</font>
<br><font size=2 face="sans-serif">srun: Warning: can't run 1 processes on 3 nodes, setting nnodes to 1</font>
<br><font size=2 face="sans-serif">I am, process 0 starting on machu139, total by srun 9</font>
<br><font size=2 face="sans-serif">I am, process 3 starting on machu139, total by srun 9</font>
<br><font size=2 face="sans-serif">I am, process 6 starting on machu139, total by srun 9</font>
<br><font size=2 face="sans-serif">I am, process 1 starting on machu140, total by srun 9</font>
<br><font size=2 face="sans-serif">I am, process 8 starting on machu141, total by srun 9</font>
<br><font size=2 face="sans-serif">I am, process 4 starting on machu140, total by srun 9</font>
<br><font size=2 face="sans-serif">I am, process 2 starting on machu141, total by srun 9</font>
<br><font size=2 face="sans-serif">I am, process 7 starting on machu140, total by srun 9</font>
<br><font size=2 face="sans-serif">I am, process 5 starting on machu141, total by srun 9</font>
<br><font size=2 face="sans-serif">Me, process 0, send 1000 to process 2</font>
<br>
<br><font size=2 face="sans-serif">...........</font>
<br>
<br><font size=2 face="sans-serif">[thipa@machu0 padb_open]$ squeue</font>
<br><font size=2 face="sans-serif"> JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)</font>
<br><font size=2 face="sans-serif"> 8324 tsl bash thipa R 36:33 3 machu[139-141]</font>
<br><font size=2 face="sans-serif">[thipa@machu0 padb_open]$ </font>
<br>
<br>
<br><font size=2 color=blue face="sans-serif"><b>padb with rmgr=slurm</b></font>
<br>
<br><font size=2 face="sans-serif">[thipa@machu0 padb_open]$ ./padb -O rmgr="slurm" -O stack-shows-locals=no -O stack-shows-params=no --debug=verbose=all -tx 8324</font>
<br><font size=2 face="sans-serif">DEBUG (verbose): 0: There are 1 processes over 3 hosts</font>
<br><font size=2 face="sans-serif">-----------------</font>
<br><font size=2 face="sans-serif">[0] (1 processes)</font>
<br><font size=2 face="sans-serif">-----------------</font>
<br><font size=2 face="sans-serif">main() at ?:?</font>
<br><font size=2 face="sans-serif"> orterun() at ?:?</font>
<br><font size=2 face="sans-serif"> opal_event_dispatch() at event.c:682</font>
<br><font size=2 face="sans-serif"> opal_event_loop() at event.c:746</font>
<br><font size=2 face="sans-serif"> poll_dispatch() at poll.c:167</font>
<br><font size=2 face="sans-serif"> poll() at ?:?</font>
<br><font size=2 face="sans-serif">DEBUG (verbose): 0: Completed command</font>
<br><font size=2 face="sans-serif">[thipa@machu0 padb_open]$ </font>
<br>
<br><font size=2 color=blue face="sans-serif"><b>padb with rmgr="sl-orte" (my patch)</b></font>
<br>
<br>
<br><font size=2 face="sans-serif">[thipa@machu0 padb_open]$ ./padb -O rmgr="sl-orte" -O stack-shows-locals=no -O stack-shows-params=no --debug=verbose=all -tx 8324</font>
<br><font size=2 face="sans-serif">DEBUG (verbose): 0: There are 1 processes over 3 hosts</font>
<br><font size=2 face="sans-serif">Warning, remote process state differs across ranks</font>
<br><font size=2 face="sans-serif">state : ranks</font>
<br><font size=2 face="sans-serif">R (running) : [2]</font>
<br><font size=2 face="sans-serif">S (sleeping) : [0-1,3-8]</font>
<br><font size=2 face="sans-serif">-----------------</font>
<br><font size=2 face="sans-serif">[0-8] (9 processes)</font>
<br><font size=2 face="sans-serif">-----------------</font>
<br><font size=2 face="sans-serif">ThreadId: 1</font>
<br><font size=2 face="sans-serif"> -----------------</font>
<br><font size=2 face="sans-serif"> [0-1,3-8] (8 processes)</font>
<br><font size=2 face="sans-serif"> -----------------</font>
<br><font size=2 face="sans-serif"> main() at pp_sndrcv_spbl.c:55</font>
<br><font size=2 face="sans-serif"> PMPI_Finalize() at pfinalize.c:46</font>
<br><font size=2 face="sans-serif"> ompi_mpi_finalize() at runtime/ompi_mpi_finalize.c:224</font>
<br><font size=2 face="sans-serif"> barrier() at grpcomm_bad_module.c:277</font>
<br><font size=2 face="sans-serif"> opal_progress() at runtime/opal_progress.c:189</font>
<br><font size=2 face="sans-serif"> ThreadId: 2</font>
<br><font size=2 face="sans-serif"> start_thread() at ?:?</font>
<br><font size=2 face="sans-serif"> btl_openib_async_thread() at btl_openib_async.c:346</font>
<br><font size=2 face="sans-serif"> poll() at ?:?</font>
<br><font size=2 face="sans-serif"> ThreadId: 3</font>
<br><font size=2 face="sans-serif"> start_thread() at ?:?</font>
<br><font size=2 face="sans-serif"> service_thread_start() at btl_openib_fd.c:427</font>
<br><font size=2 face="sans-serif"> select() at ?:?</font>
<br><font size=2 face="sans-serif"> -----------------</font>
<br><font size=2 face="sans-serif"> [2] (1 processes)</font>
<br><font size=2 face="sans-serif"> -----------------</font>
<br><font size=2 face="sans-serif"> main() at pp_sndrcv_spbl.c:50</font>
<br><font size=2 face="sans-serif"> PMPI_Recv() at precv.c:78</font>
<br><font size=2 face="sans-serif"> mca_pml_ob1_recv() at pml_ob1_irecv.c:104</font>
<br><font size=2 face="sans-serif"> opal_progress() at runtime/opal_progress.c:207</font>
<br><font size=2 face="sans-serif"> ThreadId: 2</font>
<br><font size=2 face="sans-serif"> start_thread() at ?:?</font>
<br><font size=2 face="sans-serif"> btl_openib_async_thread() at btl_openib_async.c:346</font>
<br><font size=2 face="sans-serif"> poll() at ?:?</font>
<br><font size=2 face="sans-serif"> ThreadId: 3</font>
<br><font size=2 face="sans-serif"> start_thread() at ?:?</font>
<br><font size=2 face="sans-serif"> service_thread_start() at btl_openib_fd.c:427</font>
<br><font size=2 face="sans-serif"> select() at ?:?</font>
<br><font size=2 face="sans-serif">DEBUG (verbose): 1: Completed command</font>
<br><font size=2 face="sans-serif">[thipa@machu0 padb_open]$</font>
<br>
<br>
<br><font size=2 face="sans-serif">Possibility to start jobs as follows:</font>
<br>
<br><font size=2 face="sans-serif">1-salloc ... mpirun -n 6 openmpi_appli</font>
<br><font size=2 face="sans-serif">2-salloc ....</font>
<br><font size=2 face="sans-serif"> bash: mpirun -n 6 openmpi_appli</font>
<br><font size=2 face="sans-serif">3-salloc ...</font>
<br><font size=2 face="sans-serif"> bash: srun -n 1 mpirun -n 6 openmpi_appli</font>
<br>
<br><font size=2 face="sans-serif">Here is the patch you may commit as is or work over. The patch support all possibility above.</font>
<br><font size=2 face="sans-serif">I don't use scontrol listpids, because I found this command not a universal method (some version doesn't have it),</font>
<br><font size=2 face="sans-serif">and may issued error message such as :</font>
<br><font size=2 face="sans-serif">slurmd[machu139]: proctrack/pgid does not implement slurm_container_get_pids</font>
<br>
<br><font size=2 face="sans-serif">Thipadin.</font>
<br><font size=2 face="sans-serif">More later.</font>
<br>
<br>
<br>
<br>
<br>
<br>