<br>Jie,<br>I know Ashley is away at the moment, so I will reply<br><br>By HEAD code code he means the latetst verstion of the padb source code <br><br>You could try 3.2-beta1 from <a href="http://padb.pittman.org.uk/">http://padb.pittman.org.uk/</a> dated 23-10-10<br>
<br>Hope this helps,<br>Daniel<br><br><div class="gmail_quote">On 10 November 2010 23:48, Jie Cai <span dir="ltr"><<a href="mailto:Jie.Cai@anu.edu.au">Jie.Cai@anu.edu.au</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<div class="im"><br>
On 11/11/10 06:41, Ashley Pittman wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
(2) in the PBS interactive mode of a job, I have following information and warning, please noted that no PBS job detected. I am actually expecting a pbs job detected.<br>
<br>
</blockquote>
pbs_pro support has been included for a while, pbs and Torque support are slightly different and have only been added very recently, in fact the current HEAD will detect jobs but and launch itself on the remote nodes but not find the individual processes, it is almost certainly looking for the wrong environment variable so should be easy to fix when I get some more feedback from people who are testing it (I don't have access to a pbs system and that makes it difficult).<br>
<br>
<br>
</blockquote></div>
I am pretty happy to help with this. Our PBS system is built on OpenPBS. I am not sure whether there is major difference in the interface between old OpenPBS and torque or PBS pro.<br>
<br>
Is the "HEAD" you mentioned means padb? or PBS mom? I am a little bit confused.<div class="im"><br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
$ padb --full-report=53259 --config-option rmgr=orte --rank 0padb version 3.2 (Revision 399)<br>
<br>
========<br>
Warning: errors reported by some ranks<br>
========<br>
[0]: Error message from /short/z00/jxc900/PADB/padb_build/libexec/minfo: setup_communicator_iterator() failed<br>
[0]: Stderr from minfo:<br>
WARNING: Field opal_list_next of type opal_list_item_t not found!<br>
WARNING: Field opal_list_sentinel of type opal_list_t not found!<br>
WARNING: Field fl_mpool of type ompi_free_list_t not found!<br>
WARNING: Field fl_allocations of type ompi_free_list_t not found!<br>
<br>
</blockquote>
These are errors from the MPI library, padb has done the right thing here, it's discovered the job, launched itself, found the processes but the MPI debugger callback DLL is unable to extract the information it needs. This is the second time this has been reported in as many weeks so I'm wondering if this is something that they have broken recently, the best place to take this up would be the Ompi developers list or if you can wait until after SC next week I can test if for you and report it myself.<br>
<br>
<br>
</blockquote></div>
The symbol is loaded successfully. The error happened in following code in minfo.c.<br>
<br>
res = dll_ep.setup_communicator_iterator(target_process);<br>
if ( res != mqs_ok ) {<br>
die_with_code(res,"setup_communicator_iterator() failed");<br>
}<br>
<br>
I have tested a number of OMPI versions installed on our system, from 1.3.3 to 1.4.2. All shows the warning message.<br>
<br>
I have tested padb on another cluster with 1.3.3, while no warning messages turned up.<br>
<br>
I don't know what were the configuration flags use to compile those OpenMPI libraries, and thus I don't know whether there's a potential problem that we haven't use the necessary flag to turn debug callback on (although it doesn't seem like that from OMPI available configuration list).<div class="im">
<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
That shouldn't happen, can you send the output of "gdb -p 10782" in this case?<br>
<br>
</blockquote></div>
I similar information that complaining "ptrace: operation not permitted"<br>
I did sudo for both gdb and padb:<br>
b/libopen-pal.so.0...(no debugging symbols found)...done.<br>
Loaded symbols for /apps/openmpi/1.4.2/lib/libopen-pal.so.0<br>
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.<br>
Loaded symbols for /lib64/libdl.so.2<br>
Reading symbols from /lib64/libnsl.so.1...(no debugging symbols found)...done.<br>
Loaded symbols for /lib64/libnsl.so.1<br>
Reading symbols from /lib64/libutil.so.1...(no debugging symbols found)...done.<br>
Loaded symbols for /lib64/libutil.so.1<br>
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.<br>
Loaded symbols for /lib64/libm.so.6<br>
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.<br>
[Thread debugging using libthread_db enabled]<br>
Loaded symbols for /lib64/libpthread.so.0<br>
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.<br>
Loaded symbols for /lib64/libc.so.6<br>
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.<br>
Loaded symbols for /lib64/ld-linux-x86-64.so.2<br>
Reading symbols from /apps/openmpi/1.4.2/lib/openmpi/mca_paffinity_linux.so...(no debugging symbols found)...done.<br>
Loaded symbols for /apps/openmpi/1.4.2/lib/openmpi/mca_paffinity_linux.so<br>
Reading symbols from /apps/openmpi/1.4.2/lib/openmpi/mca_carto_auto_detect.so...(no debugging symbols found)...done.<br>
Loaded symbols for /apps/openmpi/1.4.2/lib/openmpi/mca_carto_auto_detect.so<br>
Reading symbols from /apps/openmpi/1.4.2/lib/openmpi/mca_ess_hnp.so...(no debugging symbols found)...done.<br>
Loaded symbols for /apps/openmpi/1.4.2/lib/openmpi/mca_ess_hnp.so<br>
Reading symbols from /apps/openmpi/1.4.2/lib/openmpi/mca_plm_tm.so...done.<br>
Loaded symbols for /apps/openmpi/1.4.2/lib/openmpi/mca_plm_tm.so<br>
Reading symbols from /apps/openmpi/1.4.2/lib/openmpi/mca_rml_oob.so...(no debugging symbols found)...done.<br>
Loaded symbols for /apps/openmpi/1.4.2/lib/openmpi/mca_rml_oob.so<br>
Reading symbols from /apps/openmpi/1.4.2/lib/openmpi/mca_oob_tcp.so...(no debugging symbols found)...done.<br>
Loaded symbols for /apps/openmpi/1.4.2/lib/openmpi/mca_oob_tcp.so<br>
Reading symbols from /apps/openmpi/1.4.2/lib/openmpi/mca_routed_binomial.so...(no debugging symbols found)...done.<br>
Loaded symbols for /apps/openmpi/1.4.2/lib/openmpi/mca_routed_binomial.so<br>
Reading symbols from /apps/openmpi/1.4.2/lib/openmpi/mca_grpcomm_bad.so...(no debugging symbols found)...done.<br>
Loaded symbols for /apps/openmpi/1.4.2/lib/openmpi/mca_grpcomm_bad.so<br>
Reading symbols from /apps/openmpi/1.4.2/lib/openmpi/mca_ras_tm.so...(no debugging symbols found)...done.<br>
Loaded symbols for /apps/openmpi/1.4.2/lib/openmpi/mca_ras_tm.so<br>
Reading symbols from /apps/openmpi/1.4.2/lib/openmpi/mca_rmaps_round_robin.so...(no debugging symbols found)...done.<br>
Loaded symbols for /apps/openmpi/1.4.2/lib/openmpi/mca_rmaps_round_robin.so<br>
Reading symbols from /apps/openmpi/1.4.2/lib/openmpi/mca_errmgr_default.so...(no debugging symbols found)...done.<br>
Loaded symbols for /apps/openmpi/1.4.2/lib/openmpi/mca_errmgr_default.so<br>
Reading symbols from /apps/openmpi/1.4.2/lib/openmpi/mca_odls_default.so...(no debugging symbols found)...done.<br>
Loaded symbols for /apps/openmpi/1.4.2/lib/openmpi/mca_odls_default.so<br>
Reading symbols from /apps/openmpi/1.4.2/lib/openmpi/mca_iof_hnp.so...(no debugging symbols found)...done.<br>
Loaded symbols for /apps/openmpi/1.4.2/lib/openmpi/mca_iof_hnp.so<br>
Reading symbols from /apps/openmpi/1.4.2/lib/openmpi/mca_filem_rsh.so...(no debugging symbols found)...done.<br>
Loaded symbols for /apps/openmpi/1.4.2/lib/openmpi/mca_filem_rsh.so<br>
0x00007ffff6bfb14f in poll () from /lib64/libc.so.6<br>
<br>
$ sudo padb --config-option rmgr=mpirun --full-report=5226<div class="im"><br>
padb version 3.2 (Revision 399)<br></div>
full job report for job 5226<br>
<br>
Warning, failed to locate any ranks<div class="im"><br>
<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
All the patches I have are committed now, I'm still waiting for feedback on the the environment variables that pbs sets for parallel jobs however. You should try with HEAD code as at least one of your issues is fixed though (although it'll still fail further on I'm afraid).<br>
<br>
</blockquote></div>
You have mentioned a few times about HEAD code. Again, do you mean OMPI header files? or padb perl script? I will really appreciate if you could clarify this a little bit more.<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
Ashley.<br>
<br>
<br>
</blockquote>
<br>
BTW: do you have any documents, which explain how padb works, e.g work flow. It can help us significantly with understanding your code and design idea. Then we can feedback some more useful information.<br><font color="#888888">
<br>
Jie</font><div><div></div><div class="h5"><br>
<br>
_______________________________________________<br>
padb-users mailing list<br>
<a href="mailto:padb-users@pittman.org.uk" target="_blank">padb-users@pittman.org.uk</a><br>
<a href="http://pittman.org.uk/mailman/listinfo/padb-users_pittman.org.uk" target="_blank">http://pittman.org.uk/mailman/listinfo/padb-users_pittman.org.uk</a><br>
</div></div></blockquote></div><br>