From padb at googlecode.com Sun Nov 1 17:50:29 2009 From: padb at googlecode.com (padb at googlecode.com) Date: Sun, 01 Nov 2009 17:50:29 +0000 Subject: [padb-devel] [padb] r308 committed - Move the complexity for showing variables in tree based stack frames o... Message-ID: <00504502cb24965a82047752e53e@google.com> Revision: 308 Author: apittman Date: Sun Nov 1 09:49:32 2009 Log: Move the complexity for showing variables in tree based stack frames out of the display_tree function and into it's own add_data_to_tree function. This makes both the tree generation code and the complex variable reading code easier to read. There should be no change in the output generated by padb because of this commit. http://code.google.com/p/padb/source/detail?r=308 Modified: /trunk/src/padb ======================================= --- /trunk/src/padb Tue Oct 27 14:48:53 2009 +++ /trunk/src/padb Sun Nov 1 09:49:32 2009 @@ -3317,7 +3317,7 @@ # array of hashes and combine {name} and {type} into a new value # {type_name} which is the same length as {type_name} for all other entries # in the array. -sub format_local_vars { +sub _format_local_vars { my ($list) = @_; my $max = 0; @@ -3341,10 +3341,50 @@ return; } -sub _display_tree { - my ( $tree, $d, $parent, $indent, $path, $enforce_spec ) = @_; - - my $ret = $EMPTY_STRING; +sub _add_data_to_point_on_tree { + my ( $tree, $d, $l, $max_show, $peer, @vars ) = @_; + + my @all_vars; + foreach my $var (@vars) { + my @type_list = + sort keys %{ $d->{target_data}{"$peer|var_type|$var"} }; + push @all_vars, + { + name => $var, + type => $type_list[0], + }; + } + + _format_local_vars( \@all_vars ); + + foreach my $vref (@all_vars) { + my $var = $vref->{name}; + my $key = "$l|var|$var"; + my @values = keys %{ $d->{target_data}{$key} }; + + if ( @values == 1 ) { + my $line = " $vref->{type_name} = '$values[0]' " + . rng_convert_to_user( $d->{target_data}{$key}{ $values[0] } ); + push @{ $tree->{$peer}->{aux} }, $line; + } elsif ( @values > $max_show ) { + my $line = + " $vref->{type_name}: "; + push @{ $tree->{$peer}->{aux} }, $line; + } else { + push @{ $tree->{$peer}->{aux} }, " $vref->{type_name}:"; + foreach my $value ( sort @values ) { + my $line = " '$value' " + . rng_convert_to_user( $d->{target_data}{$key}{$value} ); + push @{ $tree->{$peer}->{aux} }, $line; + + } + } + } + +} + +sub _add_data_to_tree { + my ( $tree, $d, $path ) = @_; # Sort peers by lowest rank of each branch. my @peers = @@ -3354,145 +3394,101 @@ # sets this value. my $max_show = $conf{mode_options}{stack}{max_distinct_values}; - my $child_enforce_spec = 0; foreach my $peer (@peers) { - - my $vpspec = rng_convert_to_user( $tree->{$peer}->{range} ); - if ( @peers != 1 or $parent ne $vpspec or $enforce_spec ) { - $ret .= "$indent-----------------\n"; - $ret .= "$indent$vpspec ($tree->{$peer}->{count} processes)\n"; - $ret .= "$indent-----------------\n"; - } - - $ret .= "$indent$peer\n"; - if ( defined $d->{target_data} ) { - my $l = "$path,$peer"; - - if ( defined $d->{target_data}{"$peer|params"} ) { - my @params_lists = - sort keys %{ $d->{target_data}{"$peer|params"} }; - - # It's not impossible that the same function on the same - # line might have different params or locals, for example - # it could be a different binary. It's probably rare - # enough that we can ignore it however. - my @params = split $COMMA, $params_lists[0]; - - $ret .= "$indent params\n" if ( @params > 0 ); - - my @all_vars; - foreach my $var (@params) { - my @type_list = - sort keys %{ $d->{target_data}{"$peer|param_type| $var"} }; - my $type = $type_list[0]; - push @all_vars, - { - name => $var, - type => $type, - }; - } - - format_local_vars( \@all_vars ); - - foreach my $vref (@all_vars) { - my $var = $vref->{name}; - my $key = "$l|var|$var"; - my @values = keys %{ $d->{target_data}{$key} }; - my $type = $vref->{type}; - - $child_enforce_spec = 1; - if ( @values == 1 ) { - foreach my $value ( sort @values ) { - $ret .= - "$indent $vref->{type_name} = '$value' " - . rng_convert_to_user( - $d->{target_data}{$key}{$value} ) - . "\n"; - } - } elsif ( @values > $max_show ) { - $ret .= -"$indent $vref->{type_name}: \n"; - } else { - $ret .= "$indent $vref->{type_name}:\n"; - foreach my $value ( sort @values ) { - $ret .= - "$indent '$value' " - . rng_convert_to_user( - $d->{target_data}{$key}{$value} ) - . "\n"; - } - } - } + my $l = "$path,$peer"; + + if ( defined $d->{target_data}{"$peer|params"} ) { + my @params_lists = + sort keys %{ $d->{target_data}{"$peer|params"} }; + + # It's not impossible that the same function on the same + # line might have different params or locals, for example + # it could be a different binary. It's probably rare + # enough that we can ignore it however. + my @params = split $COMMA, $params_lists[0]; + + if ( @params > 0 ) { + push @{ $tree->{$peer}->{aux} }, "params"; + } + + _add_data_to_point_on_tree( $tree, $d, $l, $max_show, $peer, + @params ); + + } + + if ( defined $d->{target_data}{"$peer|locals"} ) { + my @locals_lists = + keys %{ $d->{target_data}{"$peer|locals"} }; + + # It's not impossible that the same function on the same + # line might have different params or locals, for example + # it could be a different binary. In the case of locals + # simply load all of them. + my @locals = split $COMMA, join( q{,}, @locals_lists ); + + if ( @locals > 0 ) { + push @{ $tree->{$peer}->{aux} }, "locals"; } - if ( defined $d->{target_data}{"$peer|locals"} ) { - my @locals_lists = keys %{ $d->{target_data}{"$peer| locals"} }; - - # It's not impossible that the same function on the same - # line might have different params or locals, for example - # it could be a different binary. In the case of locals - # simply load all of them. - my @locals = split $COMMA, join( q{,}, @locals_lists ); - - $ret .= "$indent locals\n" if ( @locals > 0 ); - - my @all_vars; - foreach my $var (@locals) { - my @type_list = - sort keys %{ $d->{target_data}{"$peer|var_type| $var"} }; - my $type = $type_list[0]; - push @all_vars, - { - name => $var, - type => $type, - }; - } - - format_local_vars( \@all_vars ); - - foreach my $vref (@all_vars) { - my $var = $vref->{name}; - my $key = "$l|var|$var"; - my @values = keys %{ $d->{target_data}{$key} }; - my $type = $vref->{type}; - - $child_enforce_spec = 1; - if ( @values == 1 ) { - foreach my $value ( sort @values ) { - $ret .= - "$indent $vref->{type_name} = '$value' " - . rng_convert_to_user( - $d->{target_data}{$key}{$value} ) - . "\n"; - } - } elsif ( @values > $max_show ) { - $ret .= -"$indent $vref->{type_name}: \n"; - } else { - $ret .= "$indent $vref->{type_name}:\n"; - foreach my $value ( sort @values ) { - $ret .= - "$indent '$value' " - . rng_convert_to_user( - $d->{target_data}{$key}{$value} ) - . "\n"; - } - } - } + _add_data_to_point_on_tree( $tree, $d, $l, $max_show, $peer, + @locals ); + + } + + if ( defined $tree->{$peer}->{desc} ) { + _add_data_to_tree( $tree->{$peer}->{desc}, $d, "$path,$peer" ); + } + } + return; +} + +sub add_data_to_tree { + my ( $tree, $d ) = @_; + if ( defined $d->{target_data} ) { + _add_data_to_tree( $tree, $d, $EMPTY_STRING ); + } + return; +} + +sub _display_tree { + my ( $tree, $parent, $indent, $path, $enforce_spec ) = @_; + + my $ret = $EMPTY_STRING; + + # Sort peers by lowest rank of each branch. + my @peers = + sort { $tree->{$a}->{min} <=> $tree->{$b}->{min} } keys %{$tree}; + + my $child_enforce_spec = 0; + foreach my $peer (@peers) { + + my $vpspec = rng_convert_to_user( $tree->{$peer}->{range} ); + if ( @peers != 1 or $parent ne $vpspec or $enforce_spec ) { + $ret .= "$indent-----------------\n"; + $ret .= "$indent$vpspec ($tree->{$peer}->{count} processes)\n"; + $ret .= "$indent-----------------\n"; + } + + $ret .= "$indent$peer\n"; + + if ( defined $tree->{$peer}->{aux} ) { + $child_enforce_spec = 1; + foreach my $line ( @{ $tree->{$peer}->{aux} } ) { + $ret .= "$indent $line\n"; } } + if ( defined $tree->{$peer}->{desc} ) { $ret .= _display_tree( $tree->{$peer}->{desc}, - $d, $vpspec, "$indent ", "$path,$peer", $child_enforce_spec ); + $vpspec, "$indent ", "$path,$peer", $child_enforce_spec ); } } return $ret; } sub display_tree { - my ( $tree, $d ) = @_; - return _display_tree( $tree, $d, "no-parent", $EMPTY_STRING, $EMPTY_STRING, - 1 ); + my ( $tree, ) = @_; + return _display_tree( $tree, "no-parent", $EMPTY_STRING, $EMPTY_STRING, 1 ); } # An experimental new tree format. @@ -3503,8 +3499,10 @@ foreach my $tag ( sort { $a <=> $b } keys %{$lines} ) { add_tag_to_tree( \%tree, $tag, $lines->{$tag} ); } - debug_log( 'tree', undef, 'Formatting the tree' ); - my $t = display_tree( \%tree, $d ); + debug_log( 'tree', \%tree, 'Enhancing the tree' ); + add_data_to_tree( \%tree, $d ); + debug_log( 'tree', \%tree, 'Formatting the tree' ); + my $t = display_tree( \%tree, ); debug_log( 'tree', undef, 'Displaying the tree' ); print $t; debug_log( 'tree', undef, 'Done' ); @@ -7142,7 +7140,7 @@ return unless defined $frame->{$type}; return if ( @{ $frame->{$type} } == 0 ); - format_local_vars( $frame->{$type} ); + _format_local_vars( $frame->{$type} ); output( $vp, " $type:" ); foreach my $arg ( @{ $frame->{$type} } ) { my $value = ( defined $arg->{value} ? $arg->{value} : '??' ); @@ -7348,7 +7346,7 @@ my @param_names; foreach my $par ( @{ $frame->{params} } ) { push @param_names, $par->{name}; - target_key_pair( $vp, "$l|param_type| $par->{name}", + target_key_pair( $vp, "$l|var_type| $par->{name}", $par->{type} ); if ( length $par->{value} > 70 ) { target_key_pair( From padb at googlecode.com Sun Nov 1 19:09:59 2009 From: padb at googlecode.com (padb at googlecode.com) Date: Sun, 01 Nov 2009 19:09:59 +0000 Subject: [padb] r309 committed - Add release notes for code added since the 3.0 branch was created. Message-ID: <0016e648f676dd60f60477540160@google.com> Revision: 309 Author: apittman Date: Sun Nov 1 11:09:15 2009 Log: Add release notes for code added since the 3.0 branch was created. http://code.google.com/p/padb/source/detail?r=309 Modified: /trunk/src/padb ======================================= --- /trunk/src/padb Sun Nov 1 09:49:32 2009 +++ /trunk/src/padb Sun Nov 1 11:09:15 2009 @@ -29,10 +29,40 @@ # Revision history # Version 3.? +# * Add variables to tree based stack traces. +# * Solaris port. Limited functionality compared to running on Linux +# however stack trace mode works fully. # * Add "mpirun" as a resource manager, this causes it walk the local -# process list looking for processes called mpirun and to get the pid -# and hostlist by reading data from Mpir_Proctable as specified in the -# origional paper. Padb then launches itself via pdsh. +# process list looking for processes called mpirun and to get the +# pid and hostlist by reading data from Mpir_Proctable as specified +# in the original paper. Padb then launches itself via pdsh. +# * For Open-MPI magically dip inside a number of datatypes to print +# the name as Open-MPI sees them rather than just the struct +# contents. This allows us to report communicators, datatypes and +# reduction operations by name. +# * Add a --lstopo option to run the lstopo command for each rank. +# http://www.open-mpi.org/projects/hwloc/ +# * Enhance the integration with gdb, use sequence numbers when +# talking to gdb and check that we get back what we give it. +# Correctly notice and raise an appropriate error if gdb dies +# unexpectedly. +# * Intercept pointer values for variables and instead of showing the +# pointer value show a description of what it points to using +# /proc/pid/maps +# * Nicely indent variables when showing stack traces. Indent no +# more than necessary based on the length of the variable name and +# the length of the type name. +# * Fix an error where error strings were being passed through +# sprintf, we now correctly handle errors when the error strings +# contain % characters +# * Overhauled the minfo code and the way it interacts with padb. +# Add significantly better error handling to this code. +# * Allow tracing of gdb and minfo interaction to log file. Debug +# options so require enabling at the source level. +# * Add a --create-secret-file option to generate the secret file +# automatically +# * Add SVN tags to the source file and the the revision id to the +# output of output of --version # # Version 3.0 # * Full-duplex communication between inner and outer processes, padb no @@ -51,9 +81,9 @@ # * Simplify the slurm_find_pids() function to just return the output of # scontrol listpids # * Take the old process-tree walking code from slurm_find_pids() and make -# it independant and call it for all resource managers. This allows +# it independent and call it for all resource managers. This allows # scripts which call parallel applications to be bypassed and the -# applications themselves targetted. +# applications themselves targeted. # * Added "port-range" option to limit port usage in case people try and # use padb with firewalls enabled. # @@ -219,7 +249,7 @@ # * Multi-pass argument handling, --kill also accepts --signal for example, # this should really be done at the getopt layer. Also proper usage info # for these secondary args. -# * Paramater checking of secondary args, signal has a hacky implementation +# * Parameter checking of secondary args, signal has a hacky implementation # and port-range doesn't have any checking currently. # * libunwind support? lighter weight than gdb and possibly more reliable. # * Maybe PMI would help? @@ -725,7 +755,7 @@ my %ic_names; my %ic_names_cmd; -# Debugging: this function is called periodically with a mode, an abritary +# Debugging: this function is called periodically with a mode, an arbitrary # ref and a string, it can either print simply the string or call dumper on # the ref as well. Enable with --debug=type1,type2=all my %debug_modes; @@ -804,7 +834,7 @@ next unless ( $pid =~ m{\A\d+\z}xms ); my ( undef, undef, undef, undef, $owner ) = stat "/proc/$pid"; - # Check the stat worked, it's possible for processes to dissapear + # Check the stat worked, it's possible for processes to disappear # Take care to check for defined rather than true as root has a uid # of zero. next unless defined $owner; @@ -841,7 +871,7 @@ next unless ( $pid =~ m{\A\d+\z}xms ); my ( undef, undef, undef, undef, $owner ) = stat "/proc/$pid"; - # Check the stat worked, it's possible for processes to dissapear + # Check the stat worked, it's possible for processes to disappear # Take care to check for defined rather than true as root has a uid # of zero. next unless defined $owner; @@ -2928,7 +2958,7 @@ return; } - # Multiple resource managers are installed and have jobs, bouce back to + # Multiple resource managers are installed and have jobs, bounce back to # the user to specify which one they want. print "Error, multiple active resource managers detected, use -Ormgr=\n"; @@ -3120,8 +3150,8 @@ } # Nicely format process information. XXX: proc-sort-key should probably -# sort on column headers as well as keys. Idealy we'd know what format we -# wanted and only ask the nodes to report relevent info, for now they still +# sort on column headers as well as keys. Ideally we'd know what format we +# wanted and only ask the nodes to report relevant info, for now they still # report everything. sub show_proc_format { my ( $carg, $nlines ) = @_; @@ -3811,7 +3841,7 @@ } # XXX: Should only send this list over if it makes sense, for example - # the deadlock code only works when targetting all ranks. + # the deadlock code only works when targeting all ranks. if ( defined $rank_rng ) { $req->{ranks} = $rank_rng; } @@ -4228,7 +4258,7 @@ } # Note the performance of this function is much higher when adding values -# at the top of the range than at the start, persumably it's easier to make +# at the top of the range than at the start, presumably it's easier to make # an array longer than it is to unshift something onto the start. Quietly # return if the value is already in the range. sub rng_add_value { @@ -5470,7 +5500,7 @@ # datatype. The format gdb uses is documented here: # http://sources.redhat.com/gdb/current/onlinedocs/gdb_26.html#SEC275 # -# The options $collapse argunment here is for element names that should be +# The options $collapse argument here is for element names that should be # collapsed into an array, for example in the following example each # instance if thread-id would over-write the previous one with it's own # value so to avoid this thread-ids (note the extra "s" here is passed as @@ -6468,7 +6498,7 @@ sub gdb_expand_var { my ( $gdb, $arg ) = @_; - # If you try and read a value which claims to be optimized away it + # If you try and read a value which claims to be optimised away it # will return a value of zero, hard to know how to handle this but # not reporting it is probably the better of the two options. return @@ -7159,7 +7189,7 @@ # finding main, with code to detach and try again if we don't. This served # us well on ia64 where gdb isn't very good however it as on most machines # gdb gives you results below main (__libc_start_main()) this test fails -# which causes padb to loop a number of times for each procees on a node. +# which causes padb to loop a number of times for each process on a node. # We still sometimes get garbage (due to hand-rolled memcpy()) so leave the # loop in but don't sleep every iteration. This could be handled better by # checking for the presence of one of the stack_strip_below functions in @@ -7277,7 +7307,7 @@ my $strip_below; - # Find a funtion to strip above. Only actually enable this if + # Find a function to strip above. Only actually enable this if # there is a function present which we are targeting or else no # output will be generated! Do this in reverse order so we # strip as much as possible from the stack trace. @@ -7860,7 +7890,7 @@ # Merge this reply into the local one. $handle->{child_replys}++; - # Combine the host responces. + # Combine the host responses. foreach my $status ( keys %{ $r->{host_responce} } ) { foreach my $host ( keys %{ $r->{host_responce}{$status} } ) { $handle->{all_replys}->{host_responce}{$status}{$host} = @@ -7868,7 +7898,7 @@ } } - # Combine the target process responces. + # Combine the target process responses. if ( exists $r->{target_responce} ) { foreach my $tp ( keys %{ $r->{target_responce} } ) { $handle->{all_replys}->{target_responce}{$tp} = @@ -7876,7 +7906,7 @@ } } - # Combine the target process responces from child. + # Combine the target process responses from child. if ( exists $r->{target_output} ) { foreach my $tp ( keys %{ $r->{target_output} } ) { $handle->{all_replys}->{target_output}{$tp} = @@ -7884,7 +7914,7 @@ } } - # Copy the target local responces. + # Copy the target local responses. if ( exists $handle->{target_responce} ) { foreach my $tp ( keys %{ $handle->{target_responce} } ) { $handle->{all_replys}->{target_responce}{$tp} = @@ -7899,7 +7929,7 @@ %inner_output = (); - # Copy the network target errors into responce. + # Copy the network target errors into response. if ( exists $r->{target_data} ) { if ( exists $handle->{all_replys}->{target_data} ) { foreach my $key ( keys %{ $r->{target_data} } ) { @@ -7923,7 +7953,7 @@ } } - # Merge in local target responces. + # Merge in local target responses. foreach my $key ( keys %local_target_data ) { foreach my $value ( keys %{ $local_target_data{$key} } ) { if ( defined $handle->{all_replys}->{target_data}{$key}{$value} ) { @@ -7960,7 +7990,7 @@ # Convert from a pid to a command name and do it in a safe manner to avoid # warnings. suid programs tend to have the exe link which is un-readable -# so if that yeilds nothing then load the name from the status file. +# so if that yields nothing then load the name from the status file. sub pid_to_name { my $pid = shift; my $exe = readlink "/proc/$pid/exe"; @@ -8039,7 +8069,7 @@ # The process might have died and we simply didn't find anything, # if this is the case then just skip it, the outer will notice the - # missing signon and report an approtiate error. + # missing signon and report an appropriate error. next unless defined $newpid; my $status = hash_from_status($newpid); @@ -8499,9 +8529,9 @@ sub common_main { - # The quasi-authorative list of modes padb can operate in. - - # Notes on the callback functions and paramaters. + # The quasi-authoritative list of modes padb can operate in. + + # Notes on the callback functions and parameters. # handler Called in the inner for each target process. # param: ??, $vp, $pid @@ -8511,7 +8541,7 @@ # # ??, $vp, $pid - # These two functions can eitehr return a value, and have it passed to + # These two functions can either return a value, and have it passed to # the output handler or call output() and use the # default_output_handler(). @@ -8692,7 +8722,7 @@ # These next two don't work currently pending access to a QsNet system # for testing. In the new full-duplex world startup is a little # different and these functions need updating. In particular the - # following need to be addressed. the callback paramaters are probably + # following need to be addressed. the callback parameters are probably # wrong. The shared memory key needs to be calculated. Config options # need to be read locally rather than globally $allfns{qsnet_stats} = { From padb at googlecode.com Sun Nov 1 19:13:59 2009 From: padb at googlecode.com (padb at googlecode.com) Date: Sun, 01 Nov 2009 19:13:59 +0000 Subject: [padb] r310 committed - Spelling correction: target_responce should be target_response. Message-ID: <001636e1fd88396a950477541069@google.com> Revision: 310 Author: apittman Date: Sun Nov 1 11:12:44 2009 Log: Spelling correction: target_responce should be target_response. http://code.google.com/p/padb/source/detail?r=310 Modified: /trunk/src/padb ======================================= --- /trunk/src/padb Sun Nov 1 11:09:15 2009 +++ /trunk/src/padb Sun Nov 1 11:12:44 2009 @@ -6364,8 +6364,8 @@ # XXX This is a bit of a hack to make the deadlock code work with input # files, the whole thing is due a tidy-up on the full-duplex branch # where this should be solved properly. - if ( defined $lines->{target_responce} ) { - $data = $lines->{target_responce}; + if ( defined $lines->{target_response} ) { + $data = $lines->{target_response}; } else { $data = $lines->{lines}; } @@ -7899,10 +7899,10 @@ } # Combine the target process responses. - if ( exists $r->{target_responce} ) { - foreach my $tp ( keys %{ $r->{target_responce} } ) { - $handle->{all_replys}->{target_responce}{$tp} = - $r->{target_responce}{$tp}; + if ( exists $r->{target_response} ) { + foreach my $tp ( keys %{ $r->{target_response} } ) { + $handle->{all_replys}->{target_response}{$tp} = + $r->{target_response}{$tp}; } } @@ -7915,10 +7915,10 @@ } # Copy the target local responses. - if ( exists $handle->{target_responce} ) { - foreach my $tp ( keys %{ $handle->{target_responce} } ) { - $handle->{all_replys}->{target_responce}{$tp} = - $handle->{target_responce}{$tp}; + if ( exists $handle->{target_response} ) { + foreach my $tp ( keys %{ $handle->{target_response} } ) { + $handle->{all_replys}->{target_response}{$tp} = + $handle->{target_response}{$tp}; } } @@ -7984,7 +7984,7 @@ # Reset local data. $handle->{all_replys} = undef; $handle->{child_replys} = 0; - $handle->{target_responce} = undef; + $handle->{target_response} = undef; return; } @@ -8240,7 +8240,7 @@ } else { $cargs->{out_format} = 'raw'; } - $netdata->{target_responce} = + $netdata->{target_response} = $allfns{ $cmd->{mode} }{handler_all}( $cargs, $pid_list ); 1; } or do { @@ -8271,7 +8271,7 @@ } if (%gres) { - $netdata->{target_responce} = \%gres; + $netdata->{target_response} = \%gres; } } @@ -8335,8 +8335,8 @@ if ( $netdata->{children} == 0 ) { my $res; - if ( defined $netdata->{target_responce} ) { - $res->{target_responce} = $netdata->{target_responce}; + if ( defined $netdata->{target_response} ) { + $res->{target_response} = $netdata->{target_response}; } # Save any output we've got from this node. @@ -8353,7 +8353,7 @@ # Clear down the local inputs. %inner_output = (); %local_target_data = (); - $netdata->{target_responce} = undef; + $netdata->{target_response} = undef; if ( $netdata->{shutdown} ) { inner_cleanup_and_exit($netdata); From ashley at pittman.co.uk Sun Nov 1 19:21:08 2009 From: ashley at pittman.co.uk (Ashley Pittman) Date: Sun, 01 Nov 2009 19:21:08 +0000 Subject: [padb] [padb-devel] Simple Makefile patch In-Reply-To: <20091028173250.GO53330@sun.com> References: <20091028173250.GO53330@sun.com> Message-ID: <1257103268.3430.234.camel@alpha> On Wed, 2009-10-28 at 12:32 -0500, Ethan Mallove wrote: > Little patch for Makefile to make compiling on Solaris easier. > Solaris C compiler is also named "cc", and it does not support > -Wall. Thanks, I committed this Friday. Did you see r303? it adds a (source level) option to log interaction of padb and minfo.x to a file, you could enable and send me a trace if you are having troubles with the message queues. http://code.google.com/p/padb/source/detail?r=303 Ashley, -- Ashley Pittman, Bath, UK. Padb - A parallel job inspection tool for cluster computing http://padb.pittman.org.uk From padb at googlecode.com Mon Nov 2 12:01:47 2009 From: padb at googlecode.com (padb at googlecode.com) Date: Mon, 02 Nov 2009 12:01:47 +0000 Subject: [padb] r311 committed - Extend the scope of setup_pcmd callback for resource managers, leave... Message-ID: <00504502cc5f658ccc04776224bc@google.com> Revision: 311 Author: apittman Date: Mon Nov 2 04:01:19 2009 Log: Extend the scope of setup_pcmd callback for resource managers, leave the current callback as it is but allow a new setup_job callback to replace it. setup_job differs from setup_pcmd in two ways 1) It returns it's results via a hash rather than the rather ugly list; 2) It can return a list of hosts rather than a command and have padb work out how to launch the job This is good for the mpirun resource manager as it seperates out the quering of mpirun from setting up pdsh and should also be of use to other resource managers who don't allow native launching of shadow jobs http://code.google.com/p/padb/source/detail?r=311 Modified: /trunk/src/padb ======================================= --- /trunk/src/padb Sun Nov 1 11:12:44 2009 +++ /trunk/src/padb Mon Nov 2 04:01:19 2009 @@ -358,19 +358,25 @@ # get_active_jobs user List yes Return list of all active job for user. # is_job_running job Bool no Check if a given job is running. # job_to_key job key no Convert from jobId to shm key. -# setup_pcmd job cmd|ncpus yes Command needed to launch shadow jobs. # cleanup_pcmd - - no Cleans up and temporary files. # find_pids job - maybe Called on the inner to locate pids. +# In addition one of these two is preferred setup_job has more +# flexibility however setup_pcmd is good enough for most cases. See +# the setup_jobfunction for full description. +# setup_pcmd job cmd|ncpus yes Command needed to launch shadow jobs. +# setup_job job no + # inner_rmgr var n/a no Resource manager to masquerade as. +# require_inner_callback var n/a no Resource manager doesn't preserve line +# ordering of stdout. my %rmgr; $rmgr{mpirun} = { - get_active_jobs => \&mpirun_get_jobs, - job_is_running => \&local_job_is_running, - setup_pcmd => \&mpirun_setup_pcmd, - require_inner_callback => 1, + get_active_jobs => \&mpirun_get_jobs, + job_is_running => \&local_job_is_running, + setup_job => \&mpirun_setup_job, }; $rmgr{rms} = { @@ -788,6 +794,7 @@ $debug_modes{ctree} = undef; $debug_modes{tdata} = undef; $debug_modes{config} = undef; +$debug_modes{pcmd} = undef; sub slurp_file { my ($file) = @_; @@ -2775,14 +2782,9 @@ return @jobs; } -sub mpirun_setup_pcmd { +sub mpirun_setup_job { my ($job) = @_; - if ( not find_exe('pdsh') ) { - print "mpirun resource manager requires pdsh to be installed\n"; - return; - } - my $gdb = gdb_start(); if ( not gdb_attach( $gdb, $job ) ) { if ( defined $gdb->{error} ) { @@ -2821,6 +2823,14 @@ print "No process data found"; return; } + + my %pcmd; + $pcmd{nprocesses} = $nprocs; + $pcmd{nhosts} = @hosts; + $pcmd{process_data} = \%pt; + @{ $pcmd{host_list} } = @hosts; + + return %pcmd; my $cmd = $EMPTY_STRING; if ( $hosts[0] ne hostname() or @hosts > 1 ) { @@ -2997,9 +3007,49 @@ return; } -sub setup_pcmd { +sub setup_job { my $job = shift; - return $rmgr{ $conf{rmgr} }{setup_pcmd}($job); + + # If the resource manager provides a setup_pcmd function then use it and + # simply convert the list it provides into a hash before returning it. + if ( exists $rmgr{ $conf{rmgr} }{setup_pcmd} ) { + my ( $cmd, $nprocesses, $nhosts, $pd ) = + $rmgr{ $conf{rmgr} }{setup_pcmd}($job); + my %pcmd = ( + command => $cmd, + nprocesses => $nprocesses, + nhosts => $nhosts, + process_data => $pd + ); + return %pcmd; + } + + # Otherwise call the more flexible setup_job function. + my %pcmd = $rmgr{ $conf{rmgr} }{setup_job}($job); + + # If the resource manager interface is able to give a hostlist but + # not able or willing to launch a shadow job natively then use + # pdsh to launch the inner processes. This allows us to be less + # dependant on the resource manager and work in a wider variety of + # cases. Using pdsh like this limits us to 32 hosts (More if we + # set the FANOUT pdsh environment variable) so perhaps a better + # way can be found in the future. + if ( defined $pcmd{host_list} and not defined $pcmd{command} ) { + + if ( not find_exe('pdsh') ) { + print + "$conf{rmgr} resource manager requires pdsh to be installed\n"; + return; + } + + my @hosts = @{ $pcmd{host_list} }; + if ( $hosts[0] ne hostname() or @hosts > 1 ) { + $pcmd{require_inner_callback} = 1; + my $hlist = join q{,}, @hosts; + $pcmd{command} = "pdsh -w $hlist"; + } + } + return %pcmd; } sub cleanup_pcmd { @@ -4561,12 +4611,39 @@ sub go_job { my $jobid = shift; + $conf{verbose} && print "Attaching to job $jobid\n"; + + $rem_jobid = $jobid; + + # Setup whatever is needed for running parallel commands, note this + # might involve setting environment variables. + my %pcmd = setup_job($jobid); + + debug_log( 'pcmd', \%pcmd, 'Loaded pcmd data' ); + + my $cmd = $pcmd{command}; + my $ncpus = $pcmd{nprocesses}; + my $nhosts = $pcmd{nhosts}; + my $pd = $pcmd{process_data}; + if ( defined $rmgr{ $conf{rmgr} }{require_inner_callback} and $rmgr{ $conf{rmgr} }{require_inner_callback} ) { $conf{inner_callback} = 1; } + if ( defined $pcmd{require_inner_callback} ) { + $conf{inner_callback} = $pcmd{require_inner_callback}; + } + + $conf{verbose} && defined $ncpus && print "Job has $ncpus process(es)\n"; + $conf{verbose} && defined $nhosts && print "Job spans $nhosts host(s)\n"; + + debug_log( 'verbose', undef, 'There are %d processes over %d hosts', + $ncpus, $nhosts ); + + $cmd .= " $0 --inner"; + if ( $conf{inner_callback} ) { $secret = find_padb_secret(); @@ -4578,27 +4655,11 @@ } - $conf{verbose} && print "Attaching to job $jobid\n"; - - $rem_jobid = $jobid; - - # Setup whatever is needed for running parallel commands, note this - # might involve setting environment variables. - my ( $cmd, $ncpus, $hosts, $pd ) = setup_pcmd($jobid); - - $conf{verbose} && defined $ncpus && print "Job has $ncpus process(es)\n"; - $conf{verbose} && defined $hosts && print "Job spans $hosts host(s)\n"; - - debug_log( 'verbose', undef, 'There are %d processes over %d hosts', - $ncpus, $hosts ); - - $cmd .= " $0 --inner"; - - if ( not defined $hosts ) { + if ( not defined $nhosts ) { print "Fatal problem setting up the resource manager: $conf{rmgr}\n"; return 1; } - my $errors = go_parallel( $jobid, $cmd, $ncpus, $hosts, $pd ); + my $errors = go_parallel( $jobid, $cmd, $ncpus, $nhosts, $pd ); debug_log( 'verbose', undef, 'Completed command' ); @@ -8389,6 +8450,11 @@ # it up. my $signon_text = "connect $hostname $lport $key\n"; print $signon_text; + + # Add an explicit flush here to ensure the signon is printed, + # stdout doesn't automatically get forwarded to through the + # resource manager without this here. + flush { *STDOUT }; } my $netdata; From ethan.mallove at sun.com Mon Nov 2 16:38:26 2009 From: ethan.mallove at sun.com (Ethan Mallove) Date: Mon, 2 Nov 2009 11:38:26 -0500 Subject: [padb] [padb-devel] Simple Makefile patch In-Reply-To: <1257103268.3430.234.camel@alpha> References: <20091028173250.GO53330@sun.com> <1257103268.3430.234.camel@alpha> Message-ID: <20091102163826.GC53330@sun.com> On Sun, Nov/01/2009 07:21:08PM, Ashley Pittman wrote: > On Wed, 2009-10-28 at 12:32 -0500, Ethan Mallove wrote: > > Little patch for Makefile to make compiling on Solaris easier. > > Solaris C compiler is also named "cc", and it does not support > > -Wall. > > Thanks, I committed this Friday. > > Did you see r303? it adds a (source level) option to log interaction of > padb and minfo.x to a file, you could enable and send me a trace if you > are having troubles with the message queues. > > http://code.google.com/p/padb/source/detail?r=303 > Looks like it's failing before minfo.x gets a chance to run: $ padb --debug=all --config-option rmgr=mpirun --full-report=27047 DEBUG (config): 0: Finished setting configuration options padb version 3.n (Revision 311) full job report for job 27047 DEBUG (pcmd): 0: Loaded pcmd data DEBUG (verbose): 0: There are 1 processes over 1 hosts DEBUG (verbose): 0: Remote process data available on frontend DEBUG (show_cmd): 0: /home/em162155/software/SunOS/sparc/padb/bin/padb --inner DEBUG (signon): 1: Received last signon, connecting to inner DEBUG (ctree): 1: connection tree DEBUG (full_duplex): 1: Sending command to inner, 364 bytes einner: Argument "" isn't numeric in numeric ne (!=) at /home/em162155/software/SunOS/sparc/padb/bin/padb line 8087. einner: Argument "" isn't numeric in numeric ne (!=) at /home/em162155/software/SunOS/sparc/padb/bin/padb line 8087. einner: Argument "" isn't numeric in numeric ne (!=) at /home/em162155/software/SunOS/sparc/padb/bin/padb line 8087. einner: Argument "" isn't numeric in numeric ne (!=) at /home/em162155/software/SunOS/sparc/padb/bin/padb line 8087. einner: Argument "" isn't numeric in numeric ne (!=) at /home/em162155/software/SunOS/sparc/padb/bin/padb line 8087. einner: Argument "" isn't numeric in numeric ne (!=) at /home/em162155/software/SunOS/sparc/padb/bin/padb line 8087. einner: Argument "" isn't numeric in numeric ne (!=) at /home/em162155/software/SunOS/sparc/padb/bin/padb line 8087. einner: Argument "" isn't numeric in numeric ne (!=) at /home/em162155/software/SunOS/sparc/padb/bin/padb line 8087. einner: Argument "" isn't numeric in numeric ne (!=) at /home/em162155/software/SunOS/sparc/padb/bin/padb line 8087. einner: Argument "" isn't numeric in numeric ne (!=) at /home/em162155/software/SunOS/sparc/padb/bin/padb line 8087. einner: Argument "" isn't numeric in numeric ne (!=) at /home/em162155/software/SunOS/sparc/padb/bin/padb line 8087. einner: Argument "" isn't numeric in numeric ne (!=) at /home/em162155/software/SunOS/sparc/padb/bin/padb line 8087. einner: Argument "" isn't numeric in numeric ne (!=) at /home/em162155/software/SunOS/sparc/padb/bin/padb line 8087. einner: Argument "" isn't numeric in numeric ne (!=) at /home/em162155/software/SunOS/sparc/padb/bin/padb line 8087. einner: Argument "" isn't numeric in numeric ne (!=) at /home/em162155/software/SunOS/sparc/padb/bin/padb line 8087. einner: Argument "" isn't numeric in numeric ne (!=) at /home/em162155/software/SunOS/sparc/padb/bin/padb line 8087. einner: Argument "" isn't numeric in numeric ne (!=) at /home/em162155/software/SunOS/sparc/padb/bin/padb line 8087. einner: Argument "" isn't numeric in numeric ne (!=) at /home/em162155/software/SunOS/sparc/padb/bin/padb line 8087. einner: Argument "" isn't numeric in numeric ne (!=) at /home/em162155/software/SunOS/sparc/padb/bin/padb line 8087. einner: Argument "" isn't numeric in numeric ne (!=) at /home/em162155/software/SunOS/sparc/padb/bin/padb line 8087. einner: Argument "" isn't numeric in numeric ne (!=) at /home/em162155/software/SunOS/sparc/padb/bin/padb line 8087. einner: Argument "" isn't numeric in numeric ne (!=) at /home/em162155/software/SunOS/sparc/padb/bin/padb line 8087. einner: Argument "" isn't numeric in numeric ne (!=) at /home/em162155/software/SunOS/sparc/padb/bin/padb line 8087. DEBUG (full_duplex): 1: Reply from inner, 84 bytes Warning, failed to locate any ranks DEBUG (full_duplex): 1: Sending command to inner, 28 bytes DEBUG (full_duplex): 1: Reply from inner, 84 bytes DEBUG (verbose): 1: Completed command -Ethan > Ashley, > > -- > > Ashley Pittman, Bath, UK. > > Padb - A parallel job inspection tool for cluster computing > http://padb.pittman.org.uk > From ashley at pittman.co.uk Mon Nov 2 17:09:25 2009 From: ashley at pittman.co.uk (Ashley Pittman) Date: Mon, 02 Nov 2009 17:09:25 +0000 Subject: [padb] [padb-devel] Simple Makefile patch In-Reply-To: <20091102163826.GC53330@sun.com> References: <20091028173250.GO53330@sun.com> <1257103268.3430.234.camel@alpha> <20091102163826.GC53330@sun.com> Message-ID: <1257181765.3430.334.camel@alpha> On Mon, 2009-11-02 at 11:38 -0500, Ethan Mallove wrote: > > > > http://code.google.com/p/padb/source/detail?r=303 > > > > Looks like it's failing before minfo.x gets a chance to run: > einner: Argument "" isn't numeric in numeric ne (!=) at /home/em162155/software/SunOS/sparc/padb/bin/padb line 8087. > einner: Argument "" isn't numeric in numeric ne (!=) at /home/em162155/software/SunOS/sparc/padb/bin/padb line 8087. > einner: Argument "" isn't numeric in numeric ne (!=) at /home/em162155/software/SunOS/sparc/padb/bin/padb line 8087. > DEBUG (full_duplex): 1: Reply from inner, 84 bytes > Warning, failed to locate any ranks This is the problem, in this case it's failing to find any ranks. When we had it running before there wasn't any Solaris code in get_extended_process_list() which meant it called ps many times, at least once for each process. On Linux (which can be tricked into running in Solaris mode) this slowed it down so I added code to just run ps once for all processes. Can you look at the comment at line 863 and in particular the bit about needing to check that solaris and Linux ps commands pad in the same way. This patch should show you what you need to know, there should be no undefined or empty string values in the %process_data hash. Index: padb =================================================================== --- padb (revision 311) +++ padb (working copy) @@ -8074,6 +8074,9 @@ my $ipids = $inner_conf{rmpids}; + print Dumper \%process_data; + print Dumper $ipids; + foreach my $pid ( keys %process_data ) { # The resource manager pid this pid is associated with. -- Ashley Pittman, Bath, UK. Padb - A parallel job inspection tool for cluster computing http://padb.pittman.org.uk From ethan.mallove at sun.com Mon Nov 2 17:24:26 2009 From: ethan.mallove at sun.com (Ethan Mallove) Date: Mon, 2 Nov 2009 12:24:26 -0500 Subject: [padb] [padb-devel] Simple Makefile patch In-Reply-To: <1257181765.3430.334.camel@alpha> References: <20091028173250.GO53330@sun.com> <1257103268.3430.234.camel@alpha> <20091102163826.GC53330@sun.com> <1257181765.3430.334.camel@alpha> Message-ID: <20091102172426.GD53330@sun.com> On Mon, Nov/02/2009 05:09:25PM, Ashley Pittman wrote: > On Mon, 2009-11-02 at 11:38 -0500, Ethan Mallove wrote: > > > > > > http://code.google.com/p/padb/source/detail?r=303 > > > > > > > Looks like it's failing before minfo.x gets a chance to run: > > > einner: Argument "" isn't numeric in numeric ne (!=) at /home/em162155/software/SunOS/sparc/padb/bin/padb line 8087. > > einner: Argument "" isn't numeric in numeric ne (!=) at /home/em162155/software/SunOS/sparc/padb/bin/padb line 8087. > > einner: Argument "" isn't numeric in numeric ne (!=) at /home/em162155/software/SunOS/sparc/padb/bin/padb line 8087. > > DEBUG (full_duplex): 1: Reply from inner, 84 bytes > > Warning, failed to locate any ranks > > This is the problem, in this case it's failing to find any ranks. When > we had it running before there wasn't any Solaris code in > get_extended_process_list() which meant it called ps many times, at > least once for each process. On Linux (which can be tricked into > running in Solaris mode) this slowed it down so I added code to just run > ps once for all processes. > > Can you look at the comment at line 863 and in particular the bit about > needing to check that solaris and Linux ps commands pad in the same way. > This patch should show you what you need to know, there should be no > undefined or empty string values in the %process_data hash. > > Index: padb > =================================================================== > --- padb (revision 311) > +++ padb (working copy) > @@ -8074,6 +8074,9 @@ > > my $ipids = $inner_conf{rmpids}; > > + print Dumper \%process_data; > + print Dumper $ipids; > + > foreach my $pid ( keys %process_data ) { > > # The resource manager pid this pid is associated with. > $ padb --debug=all --config-option rmgr=mpirun --full-report=27047 DEBUG (config): 0: Finished setting configuration options padb version 3.n (Revision 311) full job report for job 27047 DEBUG (pcmd): 0: Loaded pcmd data DEBUG (verbose): 0: There are 1 processes over 1 hosts DEBUG (verbose): 0: Remote process data available on frontend DEBUG (show_cmd): 0: /home/em162155/software/SunOS/sparc/padb/bin/padb --inner DEBUG (signon): 1: Received last signon, connecting to inner DEBUG (ctree): 1: connection tree DEBUG (full_duplex): 1: Sending command to inner, 368 bytes einner: Argument "" isn't numeric in numeric ne (!=) at /home/em162155/software/SunOS/sparc/padb/bin/padb line 8090. einner: Argument "" isn't numeric in numeric ne (!=) at /home/em162155/software/SunOS/sparc/padb/bin/padb line 8090. einner: Argument "" isn't numeric in numeric ne (!=) at /home/em162155/software/SunOS/sparc/padb/bin/padb line 8090. einner: Argument "" isn't numeric in numeric ne (!=) at /home/em162155/software/SunOS/sparc/padb/bin/padb line 8090. einner: Argument "" isn't numeric in numeric ne (!=) at /home/em162155/software/SunOS/sparc/padb/bin/padb line 8090. einner: Argument "" isn't numeric in numeric ne (!=) at /home/em162155/software/SunOS/sparc/padb/bin/padb line 8090. einner: Argument "" isn't numeric in numeric ne (!=) at /home/em162155/software/SunOS/sparc/padb/bin/padb line 8090. einner: Argument "" isn't numeric in numeric ne (!=) at /home/em162155/software/SunOS/sparc/padb/bin/padb line 8090. einner: Argument "" isn't numeric in numeric ne (!=) at /home/em162155/software/SunOS/sparc/padb/bin/padb line 8090. einner: Argument "" isn't numeric in numeric ne (!=) at /home/em162155/software/SunOS/sparc/padb/bin/padb line 8090. einner: Argument "" isn't numeric in numeric ne (!=) at /home/em162155/software/SunOS/sparc/padb/bin/padb line 8090. einner: Argument "" isn't numeric in numeric ne (!=) at /home/em162155/software/SunOS/sparc/padb/bin/padb line 8090. einner: Argument "" isn't numeric in numeric ne (!=) at /home/em162155/software/SunOS/sparc/padb/bin/padb line 8090. einner: Argument "" isn't numeric in numeric ne (!=) at /home/em162155/software/SunOS/sparc/padb/bin/padb line 8090. einner: Argument "" isn't numeric in numeric ne (!=) at /home/em162155/software/SunOS/sparc/padb/bin/padb line 8090. einner: Argument "" isn't numeric in numeric ne (!=) at /home/em162155/software/SunOS/sparc/padb/bin/padb line 8090. einner: Argument "" isn't numeric in numeric ne (!=) at /home/em162155/software/SunOS/sparc/padb/bin/padb line 8090. einner: Argument "" isn't numeric in numeric ne (!=) at /home/em162155/software/SunOS/sparc/padb/bin/padb line 8090. einner: Argument "" isn't numeric in numeric ne (!=) at /home/em162155/software/SunOS/sparc/padb/bin/padb line 8090. einner: Argument "" isn't numeric in numeric ne (!=) at /home/em162155/software/SunOS/sparc/padb/bin/padb line 8090. einner: Argument "" isn't numeric in numeric ne (!=) at /home/em162155/software/SunOS/sparc/padb/bin/padb line 8090. einner: Argument "" isn't numeric in numeric ne (!=) at /home/em162155/software/SunOS/sparc/padb/bin/padb line 8090. einner: Argument "" isn't numeric in numeric ne (!=) at /home/em162155/software/SunOS/sparc/padb/bin/padb line 8090. DEBUG (full_duplex): 1: Reply from inner, 84 bytes Warning, failed to locate any ranks DEBUG (full_duplex): 1: Sending command to inner, 28 bytes DEBUG (full_duplex): 1: Reply from inner, 84 bytes inner: $VAR1 = { inner: '1' => '', inner: '11888' => '', inner: '11895' => '', inner: '1746' => '1692', inner: '1752' => '1746', inner: '23506' => '', inner: '23513' => '', inner: '23970' => '', inner: '23977' => '', inner: '23979' => '', inner: '24858' => '', inner: '24865' => '', inner: '24867' => '', inner: '2595' => '1', inner: '26726' => '', inner: '26733' => '', inner: '27047' => '', inner: '27381' => '', inner: '27383' => '', inner: '27384' => '', inner: '27696' => '', inner: '27703' => '', inner: '28208' => '', inner: '28215' => '', inner: '29242' => '', inner: '29249' => '', inner: '3538' => '1', inner: '3540' => '1', inner: '3544' => '1', inner: '3548' => '1', inner: '3550' => '1', inner: '5113' => '5106', inner: '5115' => '5113', inner: '6764' => '6757', inner: '6766' => '6764', inner: '948' => '1', inner: '950' => '1', inner: '955' => '1', inner: '959' => '1' inner: }; inner: $VAR1 = { inner: '27049' => { inner: rank => '0' inner: } inner: }; DEBUG (verbose): 1: Completed command -Ethan > > -- > > Ashley Pittman, Bath, UK. > > Padb - A parallel job inspection tool for cluster computing > http://padb.pittman.org.uk > From padb at googlecode.com Mon Nov 2 17:57:09 2009 From: padb at googlecode.com (padb at googlecode.com) Date: Mon, 02 Nov 2009 17:57:09 +0000 Subject: [padb] r312 committed - Strip leading whitespace from ps output before we split it on... Message-ID: <0016e64c24f645e96d0477671bde@google.com> Revision: 312 Author: apittman Date: Mon Nov 2 09:55:58 2009 Log: Strip leading whitespace from ps output before we split it on whitespace. This ensures that the first column in the output is returned as the first list element from the split call. http://code.google.com/p/padb/source/detail?r=312 Modified: /trunk/src/padb ======================================= --- /trunk/src/padb Mon Nov 2 04:01:19 2009 +++ /trunk/src/padb Mon Nov 2 09:55:58 2009 @@ -860,11 +860,14 @@ my %procs; foreach my $proc (@procs) { - # A little bit of magic here, ps left pads its output with a - # space which the regexp matches so $pid is the second entry - # returned from the split command. I need to verify that Linux - # ps and Solaris ps behave the same in this situation. - my ( undef, $pid, $ppid ) = split $SPACE, $proc; + # A little bit of magic here, ps sometimes left pads its + # output with a space which the regexp matches. To avoid + # problems strip leadinging whitespace before we split so + # that $pid is the first entry returned from the split + # command. + $proc =~ s{\A$SPACE} + {}x; + my ( $pid, $ppid ) = split $SPACE, $proc; $procs{$pid} = $ppid; } return %procs; From ashley at pittman.co.uk Mon Nov 2 18:00:25 2009 From: ashley at pittman.co.uk (Ashley Pittman) Date: Mon, 02 Nov 2009 18:00:25 +0000 Subject: [padb] [padb-devel] Simple Makefile patch In-Reply-To: <20091102172426.GD53330@sun.com> References: <20091028173250.GO53330@sun.com> <1257103268.3430.234.camel@alpha> <20091102163826.GC53330@sun.com> <1257181765.3430.334.camel@alpha> <20091102172426.GD53330@sun.com> Message-ID: <1257184825.3430.337.camel@alpha> On Mon, 2009-11-02 at 12:24 -0500, Ethan Mallove wrote: With any luck r312 should fix this. -- Ashley Pittman, Bath, UK. Padb - A parallel job inspection tool for cluster computing http://padb.pittman.org.uk From ethan.mallove at sun.com Mon Nov 2 18:45:35 2009 From: ethan.mallove at sun.com (Ethan Mallove) Date: Mon, 2 Nov 2009 13:45:35 -0500 Subject: [padb] [padb-devel] Simple Makefile patch In-Reply-To: <1257184825.3430.337.camel@alpha> References: <20091028173250.GO53330@sun.com> <1257103268.3430.234.camel@alpha> <20091102163826.GC53330@sun.com> <1257181765.3430.334.camel@alpha> <20091102172426.GD53330@sun.com> <1257184825.3430.337.camel@alpha> Message-ID: <20091102184534.GE53330@sun.com> On Mon, Nov/02/2009 06:00:25PM, Ashley Pittman wrote: > On Mon, 2009-11-02 at 12:24 -0500, Ethan Mallove wrote: > > With any luck r312 should fix this. Success! $ padb --debug=all --config-option rmgr=mpirun --full-report=27047 DEBUG (config): 0: Finished setting configuration options padb version 3.n (Revision 312) full job report for job 27047 DEBUG (pcmd): 1: Loaded pcmd data DEBUG (verbose): 1: There are 1 processes over 1 hosts DEBUG (verbose): 1: Remote process data available on frontend DEBUG (show_cmd): 1: /home/em162155/software/SunOS/sparc/padb/bin/padb --inner DEBUG (signon): 2: Received last signon, connecting to inner DEBUG (ctree): 2: connection tree DEBUG (full_duplex): 2: Sending command to inner, 364 bytes DEBUG (full_duplex): 2: Reply from inner, 316 bytes DEBUG (full_duplex): 2: Sending command to inner, 64 bytes DEBUG (full_duplex): 3: Reply from inner, 384 bytes DEBUG (full_duplex): 3: Sending command to inner, 36 bytes DEBUG (tdata): 3: Target data Namespace: "ERROR" Error message from /home/em162155/software/SunOS/sparc/padb/bin/minfo.x: Could not find MPIR_dll_name symbol [0] Namespace: "FOUND" yes [0] Warning: errors reported by some ranks ======== [0]: Error message from /home/em162155/software/SunOS/sparc/padb/bin/minfo.x: Could not find MPIR_dll_name symbol ======== DEBUG (full_duplex): 5: Reply from inner, 432 bytes DEBUG (full_duplex): 5: Sending command to inner, 472 bytes DEBUG (tdata): 5: Target data Namespace: "ERROR" Error message from /home/em162155/software/SunOS/sparc/padb/bin/minfo.x: Could not find MPIR_dll_name symbol [0] Namespace: "FOUND" yes [0] Warning: errors reported by some ranks ======== [0]: Error message from /home/em162155/software/SunOS/sparc/padb/bin/minfo.x: Could not find MPIR_dll_name symbol ======== Total: 0 communicators, no communication data recorded. DEBUG (full_duplex): 6: Reply from inner, 1380 bytes DEBUG (full_duplex): 6: Sending command to inner, 28 bytes DEBUG (tdata): 6: Target data Namespace: ",main() at hello_c.c:18|var|argc" 1 [0] Namespace: ",main() at hello_c.c:18|var|argv" 0xffbfe2fc [0] Namespace: ",main() at hello_c.c:18|var|rank" 0 [0] Namespace: ",main() at hello_c.c:18|var|size" -4201432 [0] Namespace: "FOUND" yes [0] Namespace: "main() at hello_c.c:18|locals" rank,size [0] Namespace: "main() at hello_c.c:18|params" argc,argv [0] Namespace: "main() at hello_c.c:18|var_type|argc" int [0] Namespace: "main() at hello_c.c:18|var_type|argv" char ** [0] Namespace: "main() at hello_c.c:18|var_type|rank" int [0] Namespace: "main() at hello_c.c:18|var_type|size" int [0] DEBUG (tree): 6: Making the tree DEBUG (tree): 6: Enhancing the tree DEBUG (tree): 6: Formatting the tree DEBUG (tree): 6: Displaying the tree ----------------- [0] (1 processes) ----------------- main() at hello_c.c:18 params int argc = '1' [0] char ** argv = '0xffbfe2fc' [0] locals int rank = '0' [0] int size = '-4201432' [0] ----------------- [0] (1 processes) ----------------- sleep() at ?:? ___nanosleep() at ?:? DEBUG (tree): 6: Done DEBUG (full_duplex): 6: Reply from inner, 84 bytes DEBUG (verbose): 6: Completed command Here's the minfo.x debug log: $ cat /tmp/padb-minfo-debug-log-0-9oLDaj req: sym MPIR_dll_name ok 0xff1f9824 zzz: str:35 dmsg Could not find MPIR_dll_name symbol zzz: str:3 exit die Thanks, Ethan > > -- > > Ashley Pittman, Bath, UK. > > Padb - A parallel job inspection tool for cluster computing > http://padb.pittman.org.uk > From ashley at pittman.co.uk Mon Nov 2 19:01:52 2009 From: ashley at pittman.co.uk (Ashley Pittman) Date: Mon, 02 Nov 2009 19:01:52 +0000 Subject: [padb] [padb-devel] Simple Makefile patch In-Reply-To: <20091102184534.GE53330@sun.com> References: <20091028173250.GO53330@sun.com> <1257103268.3430.234.camel@alpha> <20091102163826.GC53330@sun.com> <1257181765.3430.334.camel@alpha> <20091102172426.GD53330@sun.com> <1257184825.3430.337.camel@alpha> <20091102184534.GE53330@sun.com> Message-ID: <1257188512.3430.343.camel@alpha> On Mon, 2009-11-02 at 13:45 -0500, Ethan Mallove wrote: > On Mon, Nov/02/2009 06:00:25PM, Ashley Pittman wrote: > > On Mon, 2009-11-02 at 12:24 -0500, Ethan Mallove wrote: > > > > With any luck r312 should fix this. > > Success! Great. You'll only be able to see the MPI message queues for a application which uses MPI, what you sent is exactly what I'd expect for your test "sleep" application. Ashley. -- Ashley Pittman, Bath, UK. Padb - A parallel job inspection tool for cluster computing http://padb.pittman.org.uk From ethan.mallove at sun.com Mon Nov 2 19:12:28 2009 From: ethan.mallove at sun.com (Ethan Mallove) Date: Mon, 2 Nov 2009 14:12:28 -0500 Subject: [padb] [padb-devel] Simple Makefile patch In-Reply-To: <1257188512.3430.343.camel@alpha> References: <20091028173250.GO53330@sun.com> <1257103268.3430.234.camel@alpha> <20091102163826.GC53330@sun.com> <1257181765.3430.334.camel@alpha> <20091102172426.GD53330@sun.com> <1257184825.3430.337.camel@alpha> <20091102184534.GE53330@sun.com> <1257188512.3430.343.camel@alpha> Message-ID: <20091102191228.GF53330@sun.com> On Mon, Nov/02/2009 07:01:52PM, Ashley Pittman wrote: > On Mon, 2009-11-02 at 13:45 -0500, Ethan Mallove wrote: > > On Mon, Nov/02/2009 06:00:25PM, Ashley Pittman wrote: > > > On Mon, 2009-11-02 at 12:24 -0500, Ethan Mallove wrote: > > > > > > With any luck r312 should fix this. > > > > Success! > > Great. You'll only be able to see the MPI message queues for a > application which uses MPI, what you sent is exactly what I'd expect for > your test "sleep" application. hello_c is an MPI application. I added the sleep loop to give me time to attach padb to it. Though I thought the minfo.x message queue stuff will not work on Solaris, as the dlopen calls haven't been ported to Solaris. -Ethan > > Ashley. > > -- > > Ashley Pittman, Bath, UK. > > Padb - A parallel job inspection tool for cluster computing > http://padb.pittman.org.uk > From ashley at pittman.co.uk Mon Nov 2 19:40:32 2009 From: ashley at pittman.co.uk (Ashley Pittman) Date: Mon, 02 Nov 2009 19:40:32 +0000 Subject: [padb] [padb-devel] Simple Makefile patch In-Reply-To: <20091102191228.GF53330@sun.com> References: <20091028173250.GO53330@sun.com> <1257103268.3430.234.camel@alpha> <20091102163826.GC53330@sun.com> <1257181765.3430.334.camel@alpha> <20091102172426.GD53330@sun.com> <1257184825.3430.337.camel@alpha> <20091102184534.GE53330@sun.com> <1257188512.3430.343.camel@alpha> <20091102191228.GF53330@sun.com> Message-ID: <1257190832.3430.379.camel@alpha> On Mon, 2009-11-02 at 14:12 -0500, Ethan Mallove wrote: > On Mon, Nov/02/2009 07:01:52PM, Ashley Pittman wrote: > > On Mon, 2009-11-02 at 13:45 -0500, Ethan Mallove wrote: > > > On Mon, Nov/02/2009 06:00:25PM, Ashley Pittman wrote: > > > > On Mon, 2009-11-02 at 12:24 -0500, Ethan Mallove wrote: > > > > > > > > With any luck r312 should fix this. > > > > > > Success! > > > > Great. You'll only be able to see the MPI message queues for a > > application which uses MPI, what you sent is exactly what I'd expect for > > your test "sleep" application. > > hello_c is an MPI application. I added the sleep loop to give me time > to attach padb to it. Though I thought the minfo.x message queue stuff > will not work on Solaris, as the dlopen calls haven't been ported to > Solaris. Ah, that's different then. It's not getting as far as the dlopen call, MPIR_dll_name is the name of a global symbol which contains the filename to dlopen. This could either mean that message queue support isn't compiled into the library or possibly that it is but some more magic is needed in the gdb_attach() function to load symbols from shared librarys as well as the actual executable, I'd assume this was working now as it's displaying stack frames from within the MPI library I assume? There doesn't appear to be anything in the output of ompi_info here to say if message queue support is enabled or not which could mean that it always is. One final thing you could try is to look for the dll itself, on my system it's in $OMPI_INSTALL_PREFIX=/lib/openmpi/libompi_dbg_msgq.so If you do have this file then you can give it's full path to padb and it won't look at the symbol for it. I have to use the option -Ompi-dll=/mnt/home/debian/ashley/code/OpenMPI/trunk-raw/install/lib/openmpi/libompi_dbg_msgq.so Ashley. -- Ashley Pittman, Bath, UK. Padb - A parallel job inspection tool for cluster computing http://padb.pittman.org.uk From ethan.mallove at sun.com Mon Nov 2 19:58:27 2009 From: ethan.mallove at sun.com (Ethan Mallove) Date: Mon, 2 Nov 2009 14:58:27 -0500 Subject: [padb] [padb-devel] Simple Makefile patch In-Reply-To: <1257190832.3430.379.camel@alpha> References: <20091028173250.GO53330@sun.com> <1257103268.3430.234.camel@alpha> <20091102163826.GC53330@sun.com> <1257181765.3430.334.camel@alpha> <20091102172426.GD53330@sun.com> <1257184825.3430.337.camel@alpha> <20091102184534.GE53330@sun.com> <1257188512.3430.343.camel@alpha> <20091102191228.GF53330@sun.com> <1257190832.3430.379.camel@alpha> Message-ID: <20091102195827.GG53330@sun.com> On Mon, Nov/02/2009 07:40:32PM, Ashley Pittman wrote: > On Mon, 2009-11-02 at 14:12 -0500, Ethan Mallove wrote: > > On Mon, Nov/02/2009 07:01:52PM, Ashley Pittman wrote: > > > On Mon, 2009-11-02 at 13:45 -0500, Ethan Mallove wrote: > > > > On Mon, Nov/02/2009 06:00:25PM, Ashley Pittman wrote: > > > > > On Mon, 2009-11-02 at 12:24 -0500, Ethan Mallove wrote: > > > > > > > > > > With any luck r312 should fix this. > > > > > > > > Success! > > > > > > Great. You'll only be able to see the MPI message queues for a > > > application which uses MPI, what you sent is exactly what I'd expect for > > > your test "sleep" application. > > > > hello_c is an MPI application. I added the sleep loop to give me time > > to attach padb to it. Though I thought the minfo.x message queue stuff > > will not work on Solaris, as the dlopen calls haven't been ported to > > Solaris. > > Ah, that's different then. It's not getting as far as the dlopen call, > MPIR_dll_name is the name of a global symbol which contains the filename > to dlopen. > > This could either mean that message queue support isn't compiled into > the library or possibly that it is but some more magic is needed in the > gdb_attach() function to load symbols from shared librarys as well as > the actual executable, I'd assume this was working now as it's > displaying stack frames from within the MPI library I assume? > > There doesn't appear to be anything in the output of ompi_info here to > say if message queue support is enabled or not which could mean that it > always is. > > One final thing you could try is to look for the dll itself, on my > system it's in $OMPI_INSTALL_PREFIX=/lib/openmpi/libompi_dbg_msgq.so If > you do have this file then you can give it's full path to padb and it > won't look at the symbol for it. I have to use the option > -Ompi-dll=/mnt/home/debian/ashley/code/OpenMPI/trunk-raw/install/lib/openmpi/libompi_dbg_msgq.so I get the same minfo.x error doing: $ padb -Ompi-dll=/ws/hpc-ct-1/hpc-ct-8.2.1/pkgs/09d/SunOS-10/sparc/built-with-sun/installs/r0Y7/install/lib/openmpi/libompi_dbg_msgq.so --debug=all --config-option rmgr=mpirun --full-report=28538 -Ethan > > Ashley. > > -- > > Ashley Pittman, Bath, UK. > > Padb - A parallel job inspection tool for cluster computing > http://padb.pittman.org.uk > From ashley at pittman.co.uk Mon Nov 2 20:23:07 2009 From: ashley at pittman.co.uk (Ashley Pittman) Date: Mon, 02 Nov 2009 20:23:07 +0000 Subject: [padb] [padb-devel] Simple Makefile patch In-Reply-To: <20091102195827.GG53330@sun.com> References: <20091028173250.GO53330@sun.com> <1257103268.3430.234.camel@alpha> <20091102163826.GC53330@sun.com> <1257181765.3430.334.camel@alpha> <20091102172426.GD53330@sun.com> <1257184825.3430.337.camel@alpha> <20091102184534.GE53330@sun.com> <1257188512.3430.343.camel@alpha> <20091102191228.GF53330@sun.com> <1257190832.3430.379.camel@alpha> <20091102195827.GG53330@sun.com> Message-ID: <1257193387.25087.24.camel@alpha> On Mon, 2009-11-02 at 14:58 -0500, Ethan Mallove wrote: > I get the same minfo.x error doing: > > $ padb -Ompi-dll=/ws/hpc-ct-1/hpc-ct-8.2.1/pkgs/09d/SunOS-10/sparc/built-with-sun/installs/r0Y7/install/lib/openmpi/libompi_dbg_msgq.so --debug=all --config-option rmgr=mpirun --full-report=28538 This error again? If so that's odd as it should have circumvented that bit of code and just loaded the dll your specified. Warning: errors reported by some ranks ======== [0]: Error message from /home/em162155/software/SunOS/sparc/padb/bin/minfo.x: Could not find MPIR_dll_name symbol Could you send me the log file for a run with that option set please. Ashley, -- Ashley Pittman, Bath, UK. Padb - A parallel job inspection tool for cluster computing http://padb.pittman.org.uk From ethan.mallove at sun.com Mon Nov 2 21:30:52 2009 From: ethan.mallove at sun.com (Ethan Mallove) Date: Mon, 2 Nov 2009 16:30:52 -0500 Subject: [padb] [padb-devel] Simple Makefile patch In-Reply-To: <1257193387.25087.24.camel@alpha> References: <20091102163826.GC53330@sun.com> <1257181765.3430.334.camel@alpha> <20091102172426.GD53330@sun.com> <1257184825.3430.337.camel@alpha> <20091102184534.GE53330@sun.com> <1257188512.3430.343.camel@alpha> <20091102191228.GF53330@sun.com> <1257190832.3430.379.camel@alpha> <20091102195827.GG53330@sun.com> <1257193387.25087.24.camel@alpha> Message-ID: <20091102213052.GH53330@sun.com> On Mon, Nov/02/2009 08:23:07PM, Ashley Pittman wrote: > On Mon, 2009-11-02 at 14:58 -0500, Ethan Mallove wrote: > > > I get the same minfo.x error doing: > > > > $ padb -Ompi-dll=/ws/hpc-ct-1/hpc-ct-8.2.1/pkgs/09d/SunOS-10/sparc/built-with-sun/installs/r0Y7/install/lib/openmpi/libompi_dbg_msgq.so --debug=all --config-option rmgr=mpirun --full-report=28538 > > This error again? If so that's odd as it should have circumvented that > bit of code and just loaded the dll your specified. > > Warning: errors reported by some ranks > ======== > [0]: Error message from /home/em162155/software/SunOS/sparc/padb/bin/minfo.x: Could not find MPIR_dll_name symbol > At least now we get a human-readble message (e.g., "Could not find MPIR_dll_name symbol"), instead of "Bad exit code 256". > Could you send me the log file for a run with that option set please. The log files are the same: $ cat /tmp/padb-minfo-debug-log-0-IkyIH9 req: sym MPIR_dll_name ok 0xff1f9824 zzz: str:35 dmsg Could not find MPIR_dll_name symbol zzz: str:3 exit die -Ethan > > Ashley, > > -- > > Ashley Pittman, Bath, UK. > > Padb - A parallel job inspection tool for cluster computing > http://padb.pittman.org.uk > From ashley at pittman.co.uk Mon Nov 2 22:06:52 2009 From: ashley at pittman.co.uk (Ashley Pittman) Date: Mon, 02 Nov 2009 22:06:52 +0000 Subject: [padb] [padb-devel] Simple Makefile patch In-Reply-To: <20091102213052.GH53330@sun.com> References: <20091102163826.GC53330@sun.com> <1257181765.3430.334.camel@alpha> <20091102172426.GD53330@sun.com> <1257184825.3430.337.camel@alpha> <20091102184534.GE53330@sun.com> <1257188512.3430.343.camel@alpha> <20091102191228.GF53330@sun.com> <1257190832.3430.379.camel@alpha> <20091102195827.GG53330@sun.com> <1257193387.25087.24.camel@alpha> <20091102213052.GH53330@sun.com> Message-ID: <1257199612.25087.54.camel@alpha> On Mon, 2009-11-02 at 16:30 -0500, Ethan Mallove wrote: > > Could you send me the log file for a run with that option set please. > > The log files are the same: > > $ cat /tmp/padb-minfo-debug-log-0-IkyIH9 > req: sym MPIR_dll_name > ok 0xff1f9824 > zzz: str:35 dmsg > Could not find MPIR_dll_name symbol > zzz: str:3 exit > die That would imply that it's found it but then not recognised it. Could you add some printfs to the find_sym() and ask() functions to see what's going on, it's the very first contact so it's likely that one of the core functions is failing, perhaps the read(0 is a problem. The attached patch should tell us more. I assume it compiles without warnings? Ashley, -- Ashley Pittman, Bath, UK. Padb - A parallel job inspection tool for cluster computing http://padb.pittman.org.uk -------------- next part -------------- A non-text attachment was scrubbed... Name: padb-minfo.patch Type: text/x-patch Size: 1192 bytes Desc: not available URL: From padb at googlecode.com Tue Nov 3 12:16:32 2009 From: padb at googlecode.com (padb at googlecode.com) Date: Tue, 03 Nov 2009 12:16:32 +0000 Subject: [padb] r313 committed - Update the minfo code to use the proposed new interface for discoverin... Message-ID: <000e0cd2113cf3ca52047776761c@google.com> Revision: 313 Author: apittman Date: Tue Nov 3 04:15:34 2009 Log: Update the minfo code to use the proposed new interface for discovering dlls. Everything should still work as before except an extra variable is checked in the target process. Also add some more error checking and report a couple of errors to the user should they occour. http://code.google.com/p/padb/source/detail?r=313 Modified: /trunk/src/minfo.c ======================================= --- /trunk/src/minfo.c Thu Oct 29 15:52:20 2009 +++ /trunk/src/minfo.c Tue Nov 3 04:15:34 2009 @@ -11,6 +11,7 @@ #include #include +#include #include "mpi_interface.h" struct dll_entry_points { @@ -108,8 +109,9 @@ #define QUERY_SIZE 1280 -/* Query (via proxy) gdb for the answer to a question, - * store the restult in ans, returns -1 on failure */ +/* Query (via proxy) gdb for the answer to a question, store the result in + * ans, returns -1 on failure + */ int ask(char *req, char *ans) { char buff[QUERY_SIZE+3]; @@ -117,8 +119,10 @@ printf("req: %s\n",req); fflush(NULL); nbytes = read(0,buff,QUERY_SIZE+3); - if ( nbytes < 0 ) + if ( nbytes < 0 ) { + show_warning("Bad result from read"); return -1; + } if ( memcmp(buff,"ok ",3 ) ) return -1; buff[nbytes-1] = '\000'; @@ -317,8 +321,9 @@ return mqs_ok; } -// Data comes back as 0x?? for each char, space required is (N*5)-1 -// 256 bytes read need 1279 bytes of space +/* Data comes back as 0x?? for each char, space required is (N*5)-1 256 + * bytes read need 1279 bytes of space + */ int find_data (mqs_process *proc, mqs_taddr_t addr, int size, void *base) { char *local = base; @@ -347,17 +352,22 @@ memcpy(b,(void *)a,size); } +/* Fetch a string from a remote memory location, making sure there is + * enough memory locally to store our copy. Return mqs_ok on success */ int fetch_string (void *bc,void *local, mqs_taddr_t remote, int size) { char req[128]; - char ans[128]; + char *ans = malloc(size+16); int i; sprintf(req,"string %d %p",size,(void *)remote); i = ask(req,ans); - if ( i != 0 ) + if ( i != 0 ) { + free(ans); return -1; + } strncpy(local,ans,size); + free(ans); return 0; } @@ -533,20 +543,98 @@ DLSYM(dll_ep,dlhandle,next_communicator); DLSYM(dll_ep,dlhandle,setup_operation_iterator); DLSYM(dll_ep,dlhandle,next_operation); - DLSYM(dll_ep,dlhandle,get_comm_group); - + + /* "Optional" symbols, these may or may not occour in the dll, if they + * are we can make use of them but if they aren't then that's fine too + */ + DLSYM_LAX(dll_ep,dlhandle,get_comm_group); DLSYM_LAX(dll_ep,dlhandle,get_global_rank); DLSYM_LAX(dll_ep,dlhandle,get_comm_coll_state); return 0; } #define PATH_MAX 1024 + +/* Try and load a valid dll from the locations array, loop over the array + * trying each one in turn. Return 0 if and when we managed to load one, + * -1 otherwise + */ +int find_and_load_dll_from_loc_array() { + void **remote_array; + char *dll_name; + void *locations = find_sym("sym","mpimsgq_dll_locations"); + + if ( locations == NULL ) + return -1; + + if ( find_data(NULL,(mqs_taddr_t)locations,sizeof(void *),&remote_array) != mqs_ok ) { + return -1; + } + + if ( (dll_name = malloc(PATH_MAX)) == NULL ) + return -1; + + do { + void *remote_entry = NULL; + + if ( find_data(NULL,(mqs_taddr_t)remote_array,sizeof(void *),&remote_entry) != mqs_ok ) + goto error_out; + + if ( remote_entry == NULL ) + goto error_out; + + memset(dll_name,0,PATH_MAX); + + if ( fetch_string(NULL,dll_name,(mqs_taddr_t)remote_entry,PATH_MAX) != mqs_ok ) { + goto error_out; + + } else { + if ( load_msgq_dll(dll_name) == 0 ) { + free(dll_name); + return mqs_ok; + } + } + remote_array++; + } while ( 1 ); + +error_out: + free(dll_name); + return -1; +} + +void find_and_load_dll() +{ + char *dll_name; + + dll_name = getenv("MPINFO_DLL"); + if ( dll_name != NULL ) { + if ( load_msgq_dll(dll_name) != 0 ) { + die("Could not load symbols from dll"); + } + return; + } + + /* Try the new (proposed) dll specification mechanism */ + if ( find_and_load_dll_from_loc_array() == mqs_ok ) + return; + + void *base = find_sym("sym","MPIR_dll_name"); + if ( base == NULL ) { + die("Could not find MPIR_dll_name symbol"); + } + dll_name = malloc(PATH_MAX); + if ( fetch_string(NULL,dll_name,(mqs_taddr_t)base,PATH_MAX) != 0 ) { + die("Could not read value of MPIR_dll_name"); + } + if ( load_msgq_dll(dll_name) != 0 ) { + die("Could not load symbols from dll"); + } +} int main () { int res; - char *dll_name; int comm_id = 0; struct image image; @@ -555,22 +643,9 @@ mqs_image *target_image = (mqs_image *)ℑ mqs_process *target_process = (mqs_process *)&process; - dll_name = getenv("MPINFO_DLL"); - if ( ! dll_name ) { - - void *base = find_sym("sym","MPIR_dll_name"); - if ( ! base ) { - die("Could not find MPIR_dll_name symbol"); - } - dll_name = malloc(PATH_MAX); - if ( fetch_string(NULL,dll_name,(mqs_taddr_t)base,PATH_MAX) != 0 ) { - die("Could not read value of MPIR_dll_name"); - } - } - - if ( load_msgq_dll(dll_name) != 0 ) { - die("Could not load symbols from dll"); - } + find_and_load_dll(); + + assert(dll_ep.setup_basic_callbacks != NULL); bcb.mqs_malloc_fp = malloc; bcb.mqs_free_fp = free; From padb at googlecode.com Tue Nov 3 13:04:00 2009 From: padb at googlecode.com (padb at googlecode.com) Date: Tue, 03 Nov 2009 13:04:00 +0000 Subject: [padb] r314 committed - Create a development branch for mpi3 tools work Message-ID: <0016e64b9a1cbb18a5047777209c@google.com> Revision: 314 Author: apittman Date: Tue Nov 3 05:03:28 2009 Log: Create a development branch for mpi3 tools work http://code.google.com/p/padb/source/detail?r=314 Added: /branches/mpi3-tools-handles From padb at googlecode.com Tue Nov 3 16:46:20 2009 From: padb at googlecode.com (padb at googlecode.com) Date: Tue, 03 Nov 2009 16:46:20 +0000 Subject: [padb] r315 committed - Import the MPI handled header file from ... Message-ID: <00504502cb1fdbe4ff04777a3b50@google.com> Revision: 315 Author: apittman Date: Tue Nov 3 08:45:14 2009 Log: Import the MPI handled header file from http://bitbucket.org/jsquyres/mpi3-tools-handles/src/tip/ompi/debuggers/mpihandles_interface.h http://code.google.com/p/padb/source/detail?r=315 Added: /branches/mpi3-tools-handles/src/mpihandles_interface.h ======================================= --- /dev/null +++ /branches/mpi3-tools-handles/src/mpihandles_interface.h Tue Nov 3 08:45:14 2009 @@ -0,0 +1,1142 @@ +/* OPEN QUESTIONS: + +- should all Fortran INTEGER references be mqs_taddr_t in case they're + not the same size as C int? + +- what to do if we have a new Fortran interface (use mpi3) with + non-INTEGER handles? + + */ + +/*---------------------------------------------------------------------------*/ + +/* + * Copyright (c) 2007 High Performance Computing Center Stuttgart, + * University of Stuttgart. All rights reserved. + * Copyright (c) 2007-2009 Cisco Systems, Inc. All rights reserved. + * Copyright (c) 2007 The University of Tennessee and The University of + * Tennessee Research Foundation. All rights reserved. + * $COPYRIGHT$ + * + * Additional copyrights may follow + * + * Some text copied from and references made to mpi_interface.h. + * + * Copyright (C) 2000-2004 by Etnus, LLC + * Copyright (C) 1999 by Etnus, Inc. + * Copyright (C) 1997-1998 Dolphin Interconnect Solutions Inc. + * + * $HEADER$ + */ + +#ifndef __MPIDBG_INTERFACE_H__ +#define __MPIDBG_INTERFACE_H__ 1 + +#include "ompi_config.h" + +/* + * This file provides interface functions for a debugger to gather + * additional information about MPI handles. + */ +#include + +/* Include the Etnus debugger message queue interface so that we can + use much of its infrastructure (e.g., the mqs_basic_callbacks, + mqs_image_callbacks, and mqs_process_callbacks). */ +#define FOR_MPI2 0 +#include "msgq_interface.h" + +/************************************************************************** + * Types and macros + **************************************************************************/ + +enum { + MPIDBG_MAX_OBJECT_NAME = MPI_MAX_OBJECT_NAME +}; +enum { + MPIDBG_MAX_FILENAME = 1024 +}; +enum { + MPIDBG_INTERFACE_VERSION = 1 +}; + + +/*----------------------------------------------------------------------- + * Global initialization information for the DLL + *-----------------------------------------------------------------------*/ + +/* Structure containing types for C and C++ MPI handles */ +struct mpidbg_handle_info_t { + /* C handle types. They are typically pointers to something or + integers. */ + /* Back-end type for MPI_Aint */ + mqs_type *hi_c_aint; + /* Back-end type for MPI_Comm */ + mqs_type *hi_c_comm; + /* Back-end type for MPI_Datatype */ + mqs_type *hi_c_datatype; + /* Back-end type for MPI_Errhandler */ + mqs_type *hi_c_errhandler; + /* Back-end type for MPI_File */ + mqs_type *hi_c_file; + /* Back-end type for MPI_Group */ + mqs_type *hi_c_group; + /* Back-end type for MPI_Info */ + mqs_type *hi_c_info; + /* Back-end type for MPI_Offset */ + mqs_type *hi_c_offset; + /* Back-end type for MPI_Op */ + mqs_type *hi_c_op; + /* Back-end type for MPI_Request */ + mqs_type *hi_c_request; + /* Back-end type for MPI_Status */ + mqs_type *hi_c_status; + /* Back-end type for MPI_Win */ + mqs_type *hi_c_win; + + /* C++ handle types. Note that these will always be *objects*, + never pointers. */ + /* Back-end type for MPI::Aint */ + mqs_type *hi_cxx_aint; + /* Back-end type for MPI::Comm */ + mqs_type *hi_cxx_comm; + /* Back-end type for MPI::Intracomm */ + mqs_type *hi_cxx_intracomm; + /* Back-end type for MPI::Intercomm */ + mqs_type *hi_cxx_intercomm; + /* Back-end type for MPI::Graphcomm */ + mqs_type *hi_cxx_graphcomm; + /* Back-end type for MPI::Cartcomm */ + mqs_type *hi_cxx_cartcomm; + /* Back-end type for MPI::Datatype */ + mqs_type *hi_cxx_datatype; + /* Back-end type for MPI::Errhandler */ + mqs_type *hi_cxx_errhandler; + /* Back-end type for MPI::File */ + mqs_type *hi_cxx_file; + /* Back-end type for MPI::Group */ + mqs_type *hi_cxx_group; + /* Back-end type for MPI::Info */ + mqs_type *hi_cxx_info; + /* Back-end type for MPI::Offset */ + mqs_type *hi_cxx_offset; + /* Back-end type for MPI::Op */ + mqs_type *hi_cxx_op; + /* Back-end type for MPI::Request */ + mqs_type *hi_cxx_request; + /* Back-end type for MPI::Prequest */ + mqs_type *hi_cxx_prequest; + /* Back-end type for MPI::Grequest */ + mqs_type *hi_cxx_grequest; + /* Back-end type for MPI::Status */ + mqs_type *hi_cxx_status; + /* Back-end type for MPI::Win */ + mqs_type *hi_cxx_win; +}; + +enum mpidbg_return_codes_t { + /* Success */ + MPIDBG_SUCCESS, + /* Something was not found */ + MPIDBG_ERR_NOT_FOUND, + /* Something is not supported */ + MPIDBG_ERR_NOT_SUPPORTED, + /* Something is out of range */ + MPIDBG_ERR_OUT_OF_RANGE, + /* Something is not available */ + MPIDBG_ERR_UNAVAILABLE, + /* Ran out of memory */ + MPIDBG_ERR_NO_MEM, + /* Sentinel max value */ + MPIDBG_MAX_RETURN_CODE +}; + +/*----------------------------------------------------------------------- + * General data structures + *-----------------------------------------------------------------------*/ + +/* Information about MPI processes */ +struct mpidbg_process_t { + /* JMS: need something to uniquely ID MPI processes in the + presence of MPI_COMM_SPAWN */ + + /* Global rank in MPI_COMM_WORLD */ + int mpi_comm_world_rank; +}; +/* ==> JMS Should we just use mqs_process_location instead? George + thinks that this is unncessary -- perhaps due to the fact that we + could use mqs_process_location...? Need to get some feedback from + others on this one. Need to check Euro PVM/MPI '06 paper... */ + +/* General name -> handle address mappings. This is an optional type + that is used to describe MPI's predefined handles if the + pre-defined names do not appear as symbols in the MPI process. + E.g., if MPI_COMM_WORLD is a #define that maps to some other value, + this data structure can be used to map the string "MPI_COMM_WORLD" + to the actual value of the handle that it corresponds to (e.g., 0 + or a pointer value). */ +struct mpidbg_name_map_t { + /* Name of the handle */ + char *map_name; + + /* Handle that the name corresponds to. Will be 0/NULL if there + is no corresponding back-end object. */ + mqs_taddr_t map_handle; +}; + +/* MPI attribute / value pairs. Include both a numeric and string + key; pre-defined MPI keyvals (e.g., MPI_TAG_MAX) have a + human-readable string name. The string will be NULL for + non-predefined keyvals. + + The int keyval member is last to avoid "holes" in the memory + layout. */ +struct mpidbg_attribute_pair_t { + /* Keyval name; will be non-NULL for attributes that have a + human-readable name (i.e., MPI predefined keyvals) */ + char *keyval_name; + /* Value */ + char *value; + /* Keyval */ + int keyval; +}; + +/*----------------------------------------------------------------------- + * Communicators + *-----------------------------------------------------------------------*/ + +/* Using an enum instead of #define because debuggers can show the + *names* of enum values, not just the values. */ +enum mpidbg_comm_capabilities_t { + /* Whether this MPI DLL supports returning basic information about + communicators */ + MPIDBG_COMM_CAP_BASIC = 0x01, + /* Whether this MPI DLL supports returning names of + communicators */ + MPIDBG_COMM_CAP_STRING_NAMES = 0x02, + /* Whether this MPI DLL supports indicating whether a communicator + has been freed by the user application */ + MPIDBG_COMM_CAP_FREED_HANDLE = 0x04, + /* Whether this MPI DLL supports indicating whether a communicator + object has been freed by the MPI implementation or not */ + MPIDBG_COMM_CAP_FREED_OBJECT = 0x08, + /* Whether this MPI DLL supports returning the list of MPI request + handles that are pending on a communicator */ + MPIDBG_COMM_CAP_REQUEST_LIST = 0x10, + /* Whether this MPI DLL supports returning the list of MPI window + handles that were derived from a given communicator */ + MPIDBG_COMM_CAP_WINDOW_LIST = 0x20, + /* Whether this MPI DLL supports returning the list of MPI file + handles that were derived from a given communicator */ + MPIDBG_COMM_CAP_FILE_LIST = 0x40, + /* Sentinel max value */ + MPIDBG_COMM_CAP_MAX +}; + +enum mpidbg_comm_info_bitmap_t { + /* Predefined communicator if set (user-defined if not set) */ + MPIDBG_COMM_INFO_PREDEFINED = 0x001, + /* Whether this communicator is a cartesian communicator or not + (mutually exclusive with _GRAPH and _INTERCOMM) */ + MPIDBG_COMM_INFO_CARTESIAN = 0x002, + /* Whether this communicator is a graph communicator or not + (mutually exclusive with _CARTESIAN and _INTERCOMM) */ + MPIDBG_COMM_INFO_GRAPH = 0x004, + /* If a cartesian or graph communicator, whether the processes in + this communicator were re-ordered when the topology was + assigned. */ + MPIDBG_COMM_INFO_TOPO_REORDERED = 0x008, + + /* Whether this is an intercommunicator or not (this communicator + is an intracommunicator if this flag is not yet). */ + MPIDBG_COMM_INFO_INTERCOMM = 0x010, + + /* This communicator has been marked for freeing by the user + application if set */ + MPIDBG_COMM_INFO_FREED_HANDLE = 0x020, + /* This communicator has actually been freed by the MPI + implementation if set */ + MPIDBG_COMM_INFO_FREED_OBJECT = 0x040, + /* The queried communicator is MPI_COMM_NULL */ + MPIDBG_COMM_INFO_COMM_NULL = 0x080, + + /* The queried communicator is a C handle */ + MPIDBG_COMM_INFO_HANDLE_C = 0x100, + /* The queried communicator is a C++ handle */ + MPIDBG_COMM_INFO_HANDLE_CXX = 0x200, + /* The queried communicator is a F77/F90 integer handle */ + MPIDBG_COMM_INFO_HANDLE_FINT = 0x400, + + /* Sentinel max value */ + MPIDBG_COMM_INFO_MAX +}; + +/* Predefined handle -> address mappings. This enum is the index to + an array indicating the handle addresses of the 4 pre-defined + communicators. */ +enum mpidbg_predefined_comm_t { + MPIDBG_COMM_WORLD, + MPIDBG_COMM_SELF, + MPIDBG_COMM_PARENT, + MPIDBG_COMM_NULL, + MPIDBG_COMM_MAX +}; + +/* When a communicator is looked up in an MPI process, the following + handle is returned. This handle can be used as a "base class" by + the DLL to cache additional information, if desired. This handle + is a distinct type (rather than, for example, a typedef to (void*)) + to provide compile-time checking, ensuring that handles are not + queried from one type and used with another. */ +struct mpidbg_comm_handle_t { + /* Image that this handle is in */ + mqs_image *image; + mqs_image_info *image_info; + + /* Process that this handle is in */ + mqs_process *process; + mqs_process_info *process_info; + + /* Value of the communicator handle in the MPI process (passed in + via mpidbg_comm_query()) */ + mqs_taddr_t c_comm; +}; + +/*----------------------------------------------------------------------- + * Requests + *-----------------------------------------------------------------------*/ + +/* Using an enum instead of #define because debuggers can show the + *names* of enum values, not just the values. */ +enum mpidbg_request_capabilities_t { + /* Whether this MPI DLL supports returning basic information about + requests */ + MPIDBG_REQUEST_CAP_BASIC = 0x01, + /* Sentinel max value */ + MPIDBG_REQUEST_CAP_MAX +}; + +enum mpidbg_request_info_bitmap_t { + /* Predefined request if set (user-defined if not set) */ + MPIDBG_REQUEST_INFO_PREDEFINED = 0x01, + /* Sentinel max value */ + MPIDBG_REQUEST_INFO_MAX +}; + +/* See note about mpidbg_comm_handle_t, above */ +struct mpidbg_request_handle_t { + /* Image that this handle is in */ + mqs_image *image; + mqs_image_info *image_info; + + /* Process that this handle is in */ + mqs_process *process; + mqs_process_info *process_info; + + /* Value of the request handle in the MPI process (passed in via + mpidbg_request_query()) */ + mqs_taddr_t c_request; +}; + +/*----------------------------------------------------------------------- + * Statuses + *-----------------------------------------------------------------------*/ + +enum mpidbg_status_capabilities_t { + /* Whether this MPI DLL supports returning basic information about + statuses */ + MPIDBG_STATUS_CAP_BASIC = 0x01, + /* Sentinel max value */ + MPIDBG_STATUS_CAP_MAX +}; + +enum mpidbg_status_info_bitmap_t { + /* Predefined status if set (user-defined if not set) */ + MPIDBG_STATUS_INFO_PREDEFINED = 0x01, + /* Sentinel max value */ + MPIDBG_STATUS_INFO_MAX +}; + +/* See note about mpidbg_comm_handle_t, above */ +struct mpidbg_status_handle_t { + /* Image that this handle is in */ + mqs_image *image; + mqs_image_info *image_info; + + /* Process that this handle is in */ + mqs_process *process; + mqs_process_info *process_info; + + /* Value of the status handle in the MPI process (passed in + via mpidbg_status_query()) */ + mqs_taddr_t c_status; +}; + +/*----------------------------------------------------------------------- + * Error handlers + *-----------------------------------------------------------------------*/ + +/* Using an enum instead of #define because debuggers can show the + *names* of enum values, not just the values. */ +enum mpidbg_errhandler_capabilities_t { + /* Whether this MPI DLL supports returning basic information about + error handlers */ + MPIDBG_ERRH_CAP_BASIC = 0x01, + /* Whether this MPI DLL supports returning names of the predefined + error handlers */ + MPIDBG_ERRH_CAP_STRING_NAMES = 0x02, + /* Whether this MPI DLL supports indicating whether an error + handler has been freed by the user application */ + MPIDBG_ERRH_CAP_FREED_HANDLE = 0x04, + /* Whether this MPI DLL supports indicating whether an error + handler object has been freed by the MPI implementation or + not */ + MPIDBG_ERRH_CAP_FREED_OBJECT = 0x08, + /* Whether this MPI DLL supports returning the list of MPI handles + that an MPI error handler is attached to */ + MPIDBG_ERRH_CAP_HANDLE_LIST = 0x10, + /* Sentinel max value */ + MPIDBG_ERRH_CAP_MAX +}; + +enum mpidbg_errhandler_info_bitmap_t { + /* Predefined error handler if set (user-defined if not set) */ + MPIDBG_ERRH_INFO_PREDEFINED = 0x01, + /* Communicator error handler if set */ + MPIDBG_ERRH_INFO_COMMUNICATOR = 0x02, + /* File error handler if set */ + MPIDBG_ERRH_INFO_FILE = 0x04, + /* Window error handler if set */ + MPIDBG_ERRH_INFO_WINDOW = 0x08, + /* Callback is in C if set */ + MPIDBG_ERRH_INFO_C_CALLBACK = 0x10, + /* Callback is in Fortran if set */ + MPIDBG_ERRH_INFO_FORTRAN_CALLBACK =0x20, + /* Callback is in C++ if set */ + MPIDBG_ERRH_INFO_CXX_CALLBACK = 0x40, + /* This errorhandler has been marked for freeing by the user + application if set */ + MPIDBG_ERRH_INFO_FREED_HANDLE = 0x80, + /* This errorhandler has actually been freed by the MPI + implementation if set */ + MPIDBG_ERRH_INFO_FREED_OBJECT = 0x100, + /* Sentinel max value */ + MPIDBG_ERRH_INFO_MAX +}; + +/* Predefined handle -> address mappings. This enum is the index to + an array indicating the handle addresses of the pre-defined + errhandlers. */ +enum mpidbg_predefined_errhandler_t { + MPIDBG_ERRHANDLER_ARE_FATAL, + MPIDBG_ERRHANDLER_RETURN, + MPIDBG_ERRHANDLER_THROW_EXCEPTIONS, + MPIDBG_ERRHANDLER_NULL, + MPIDBG_ERRHANDLER_MAX +}; + +/* See note about mpidbg_comm_handle_t, above */ +struct mpidbg_errhandler_handle_t { + /* Image that this handle is in */ + mqs_image *image; + mqs_image_info *image_info; + + /* Process that this handle is in */ + mqs_process *process; + mqs_process_info *process_info; + + /* Value of the errhandler handle in the MPI process (passed in + via mpidbg_errhandler_query()) */ + mqs_taddr_t c_errhandler; +}; + +/************************************************************************** + * Global variables + * + * mpidbg_dll_locations is in the MPI application; all others are in + * the DLL. + **************************************************************************/ + +/* Array of filenames instantiated IN THE MPI APPLICATION (*NOT* in + the DLL) that provides an set of locations where DLLs may be found. + The last pointer in the array will be a NULL sentinel value. The + debugger can scan the entries in the array, find one that matches + the debugger (i.e., whether the dlopen works or not), and try to + dynamically open the dl_filename. Notes: + + 1. It is not an error if a dl_filename either does not exist or is + otherwise un-openable (the debugger can just try the next + match). + 2. This array values are not valid until MPIR_Breakpoint. + 3. If a filename is absolute, the debugger will attempt to load + exactly that. If the filename is relative, the debugger may try + a few prefix variations to find the DLL. + */ +extern char **mpidbg_dll_locations; + +/* Global variable *in the DLL* describing the DLL's capabilties with + regards to communicators. This value is valid after a successfull + call to mpidbg_init_per_process(). */ +extern enum mpidbg_comm_capabilities_t mpidbg_comm_capabilities; + +/* Global variable *in the DLL* that is an array of MPI communicator + handle names -> handle mappings (the last entry in the array is + marked by a NULL string value). For example, MPI_COMM_WORLD may + not appear as a symbol in an MPI process, but the debugger needs to + be able to map this name to a valid handle. MPI implementations + not requiring this mapping can either have a NULL value for this + variable or have a single entry that has a NULL string value. This + variable is not valid until after a successfull call to + mpidbg_init_per_process(). */ +extern mqs_taddr_t mpidbg_predefined_comm_map[MPIDBG_COMM_MAX]; + +/* Global variable *in the DLL* describing the DLL's capabilties with + regards to error handlers. This value is valid after a successfull + call to mpidbg_init_per_process(). */ +extern enum mpidbg_errhandler_capabilities_t mpidbg_errhandler_capabilities; + +/* Global variable *in the DLL* that is an array of MPI error handler + handle names -> handle mappings. It is analogous to + mpidbg_predefined_comm_map; see above for details. */ +extern mqs_taddr_t mpidbg_predefined_errhandler_map[MPIDBG_ERRHANDLER_MAX]; + +/************************************************************************** + * Functions + **************************************************************************/ + +/*----------------------------------------------------------------------- + * DLL infrastructure functions + *-----------------------------------------------------------------------*/ + +/* This function must be called once before any other mpidbg_*() + function is called, and before most other global mpidbg_* data is + read. It is only necessary to call this function once for a given + debugger instantiation. This function will initialize all mpidbg + global state, to include setting all relevant global capability + flags. + + Parameters: + + IN: callbacks: Table of pointers to the debugger functions. The DLL + need only save the pointer, the debugger promises to + maintain the table of functions valid for as long as + needed. The table remains the property of the + debugger, and should not be altered or deallocated + by the DLL. This applies to all of the callback + tables. + + This function will return: + + MPIDBG_SUCCESS: if all initialization went well + MPIDBG_ERR_*: if something went wrong. +*/ +int mpidbg_init_once(const mqs_basic_callbacks *callbacks); + +/*-----------------------------------------------------------------------*/ + +/* Query the DLL to find out what version of the interface it + supports. + + Parameters: + + None. + + This function will return: + + MPIDBG_INTERFACE_VERSION +*/ + +int mpidbg_interface_version_compatibility(void); + +/*-----------------------------------------------------------------------*/ + +/* Returns a string describing this DLL. + + Parameters: + + None + + This function will return: + + A null-terminated string describing this DLL. +*/ +char *mpidbg_version_string(void); + +/*-----------------------------------------------------------------------*/ + +/* Returns the address width that this DLL was compiled with. + + Parameters: + + None + + This function will return: + + sizeof(mqs_taddr_t) +*/ + +int mpidbg_dll_taddr_width(void); + +/*-----------------------------------------------------------------------*/ + +/* Setup debug information for a specific image, this must save the + callbacks (probably in the mqs_image_info), and use those functions + for accessing this image. + + The DLL should use the mqs_put_image_info and mqs_get_image_info + functions to associate whatever information it wants to keep with + the image (e.g., all of the type offsets it needs could be kept + here). The debugger will call mqs_destroy_image_info when it no + longer wants to keep information about the given executable. + + This will be called once for each executable image in the parallel + job. + + Parameters: + + IN: image: the application image. + IN: callbacks: Table of pointers to the debugger image-specific + functions. The DLL need only save the pointer, the + debugger promises to maintain the table of functions + valid for as long as needed. The table remains the + property of the debugger, and should not be altered + or deallocated by the DLL. This applies to all of + the callback tables. + IN/OUT: handle_types: a pointer to a pre-allocated struct + containing mqs_types for each of the MPI + handle types. Must be filled in with results + from mqs_find_type for each MPI handle type. + + This function will return: + + MPIDBG_SUCCESS: if all initialization went well + MPIDBG_ERR_NOT_SUPPORTED: if the image does not support the MPIDBG + interface. In this case, no other mpidbg functions + will be invoked on this image (not even + mpidbg_finalize_per_image()). + MPIDBG_ERR_*: if something went wrong. +*/ +int mpidbg_init_per_image(mqs_image *image, + const mqs_image_callbacks *callbacks, + struct mpidbg_handle_info_t *handle_types); + +/* This function will be called once when an application image that + previously had mpidbg_init_per_image() successfully invoked that is + now ending (e.g., the debugger is exiting, the debugger has + unloaded this image, etc.). This function can be used to clean up + any image-specific data. + + Parameters: + + IN: image: the application image. + IN: image_info: the info associated with the application image. +*/ +void mpidbg_finalize_per_image(mqs_image *image, mqs_image_info *image_info); + +/*-----------------------------------------------------------------------*/ + +/* This function will only be called if mpidbg_init_per_image() + returned successfully, indicating that the image contains + information for MPI handle information. If you cannot tell whether + a process will have MPI handle information in it by examining the + image, you should return SUCCESS from mpidbg_init_per_image() and + use this function to check whether MPI handle information is + available in the process. + + Set up whatever process specific information we need. For instance, + addresses of global variables should be handled here rather than in + the image information, because if data may be in dynamic libraries + which could end up mapped differently in different processes. + + Note that certain global variables are not valid until after this + call completes successfully (see above; e.g., + mpidbg_comm_capabilities, mpidbg_comm_name_mapping, etc.). + + Parameters: + + IN: process: the process + IN: callbacks: Table of pointers to the debugger process-specific + functions. The DLL need only save the pointer, the + debugger promises to maintain the table of functions + valid for as long as needed. The table remains the + property of the debugger, and should not be altered + or deallocated by the DLL. This applies to all of + the callback tables. + IN/OUT: handle_types: the same handle_types that was passed to + mqs_init_per_image. It can be left unaltered + if the results from mqs_init_per_image were + sufficient, or modified if necessary to be + specific to this process. + + This function will return: + + MPIDBG_SUCCESS: if all initialization went well + MPIDBG_ERR_NOT_SUPPORTED: if the process does not support the MPIDBG + interface. In this case, no other mpidbg functions + will be invoked on this image (not even + mpidbg_finalize_per_process()). + MPIDBG_ERR_*: if something went wrong. +*/ +int mpidbg_init_per_process(mqs_process *process, + const mqs_process_callbacks *callbacks, + struct mpidbg_handle_info_t *handle_types); + +/* This function will be called once when an application image that + previously had mpidbg_init_per_process() successfully invoked that + is now ending (e.g., the debugger is exiting, the debugger has + stopped executing this process, etc.). This function can be used + to clean up any process-specific data. + + Parameters: + + IN: process: the application process. + IN: process_info: the info associated with the application process. +*/ +void mpidbg_finalize_per_process(mqs_process *process, + mqs_process_info *process_info); + +/*----------------------------------------------------------------------- + * MPI handle query functions + * MPI_Comm + *-----------------------------------------------------------------------*/ + +/* Query a specific MPI_Comm handle and, if found and valid, return a + handle that can subsequently be queried for a variety of + information about this communicator. + + Note that the returned handle is only valid at a specific point in + time. If the MPI process advances after the handle is returned, + the handle should be considered stale and should therefore be + freed. This function should be invoked again to obtain a new + handle. + + The intent behind this design (query the communicator once and + return a handle for subsequent queries) is to allow a DDL to scan + the MPI process *once* for all the relevant information about the + communicator and cache it locally (presumably on the returned + handle). The subsequent queries are then all local, not involving + the probing the MPI process -- which chould be somewhat cheaper / + faster / easier to implement. + + Of course, a DLL is free to cache only the communicator value in + the handle and actually probe the MPI process in any of the + subsequent query functions, if desired. To be clear: this design + *allows* for the pre-caching of all the MPI process communicator + data, but does not mandate it. + + Upon successful return (i.e., returning MPIDBG_SUCCESS), the + returned comm_handle->c_comm will equal the c_comm parameter value. + The caller must also eventually free the returned handle via + mpidbg_comm_handle_free(). + + Parameters: + + IN: image: image + IN: image_info: image info that was previously "put" + IN: process: process + IN: process_info: process info that was previously "put" + IN: comm: communicator handle + OUT: comm_handle: handle to be passed to the query functions (below) + + This function will return: + + MPIDBG_SUCCESS: if the communicator handle is valid, was found, and + the OUT parameter was filled in successfully. + MPIDBG_ERR_NOT_FOUND: if the handle is not valid / found. + MPIDBG_ERR_UNSUPPORTED: if this function is unsupported. +*/ +int mpidbg_comm_query(mqs_image *image, + mqs_image_info *image_info, + mqs_process *process, + mqs_process_info *process_info, + mqs_taddr_t comm, + struct mpidbg_comm_handle_t **handle); + +/* Free a handle returned by the mpidbg_comm_query() function. + + Parameters: + + IN: handle: handle previously returned by mpidbg_comm_query() + + This function will return: + + MPIDBG_SUCCESS: if the handle is valid and was freed successfully. + MPIDBG_ERR_NOT_FOUND: if the handle is not valid / found. +*/ +int mpidbg_comm_handle_free(struct mpidbg_comm_handle_t *handle); + +/* Query a handle returned by mpidbg_comm_query() and, if found and + valid, return basic information about the original communicator. + + Parameters: + + IN: comm_handle: handle returned by mpidbg_comm_query() + OUT: comm_name: string name of the communicator, max of + MPIDBG_MAX_OBJECT_NAME characters, \0-terminated if less than + MPIDBG_MAX_OBJECT_NAME characters. + OUT: comm_bitflags: bit flags describing the communicator + OUT: comm_rank: rank of this process in this communicator + OUT: comm_size: total number of processes in this communicator + OUT: comm_fortran_handle: INTEGER Fortran handle corresponding to + this communicator, or MPIDBG_ERR_UNAVAILABLE if currently + unavailable, or MPIDBG_ERR_NOT_SUPPORTED if not supported + OUT: comm_cxx_handle: Pointer to C++ handle corresponding to + this communicator, or MPIDBG_ERR_UNAVAILABLE if currently + unavailable, or MPIDBG_ERR_NOT_SUPPORTED if not supported + + This function will return: + + MPIDBG_SUCCESS: if the handle is valid, was found, and the OUT + parameters were filled in successfully. + MPIDBG_ERR_NOT_FOUND: if the handle is not valid / found. + MPIDBG_ERR_UNSUPPORTED: if this function is unsupported. +*/ +int mpidbg_comm_query_basic(struct mpidbg_comm_handle_t *handle, + char comm_name[MPIDBG_MAX_OBJECT_NAME], + enum mpidbg_comm_info_bitmap_t *comm_bitflags, + int *comm_rank, + int *comm_size, + int *comm_fortran_handle, + mqs_taddr_t *comm_cxx_handle); + +/* Query a handle returned by mpidbg_comm_query() and, if found and + valid, return information about the MPI processes in this + communicator. + + All arrays returned in OUT variables are allocated by this + function, but are the responsibility of the caller to be freed. + + Parameters: + + IN: comm_handle: handle returned by mpidbg_comm_query() + OUT: comm_num_local_procs: filled with the length of the + comm_local_procs OUT array. + OUT: comm_local_procs: filled with a pointer to an array of + mpidbg_process_t instances describing the local processes in + this communicator. + OUT: comm_num_remote_procs: filled with the length of the + comm_remote_procs OUT array. Will be 0 if the communicator is + not an intercommunicator. + OUT: comm_remote_procs: filled with a pointer to an array of + mpidbg_process_t instances describing the local processes in + this communicator. Will be NULL if the communicator is not an + intercommunicator. + + This function will return: + + MPIDBG_SUCCESS: if the handle is valid, was found, and the OUT + parameters were filled in successfully. + MPIDBG_ERR_NOT_FOUND: if the handle is not valid / found. + MPIDBG_ERR_UNSUPPORTED: if this function is unsupported. +*/ +int mpidbg_comm_query_procs(struct mpidbg_comm_handle_t *handle, + int *comm_num_local_procs, + struct mpidbg_process_t **comm_local_procs, + int *comm_num_remote_procs, + struct mpidbg_process_t **comm_remote_procs); + +/* Query a handle returned by mpidbg_comm_query() and, if found and + valid, return information about the topology associated with the + communicator (if any). It is not an error to call this function + with communicators that do not have associated topologies; such + communicators will still return MPIDBG_SUCCESS but simply return a + value of 0 in the comm_out_length OUT parameter. + + All arrays returned in OUT variables are allocated by this + function, but are the responsibility of the caller to be freed. + + Parameters: + + IN: comm_handle: handle returned by mpidbg_comm_query() + OUT: comm_out_length: + - For cartesian communicators, filled with the number of + dimensions. + - For graph communicators, filled with the number of nodes. + - For all other communicators, filled with 0. + OUT: comm_cart_dims_or_graph_indexes: + - For cartesian communicators, filled with a pointer to an + array of length *comm_out_length representing the dimenstion + lengths. + - For graph communicators, filled with a pointer to an array + of length *comm_out_length representing the node degrees. + - For all other communicators, filled with NULL. + OUT: comm_cart_periods_or_graph_edges: + - For cartesian communicators, filled with a pointer to an + array of length *comm_out_length representing whether each + dimension is periodic or not. + - For graph communicators, filled with a pointer to an array + of length *comm_out_length representing the array of edges. + - For all other communicators, filled with NULL. + + This function will return: + + MPIDBG_SUCCESS: if the handle is valid, was found, and the OUT + parameters were filled in successfully. + MPIDBG_ERR_NOT_FOUND: if the handle is not valid / found. + MPIDBG_ERR_UNSUPPORTED: if this function is unsupported. +*/ +int mpidbg_comm_query_topo(struct mpidbg_comm_handle_t *handle, + int *comm_out_length, + int **comm_cart_dims_or_graph_indexes, + int **comm_cart_periods_or_graph_edges); + +/* Query a handle returned by mpidbg_comm_query() and, if found and + valid, return information about the attributes associated with the + communicator (if any). It is not an error to call this function + with communicators that do not have associated attributes; such + communicators will still return MPIDBG_SUCCESS but simply return a + value of 0 in the comm_attrs_length OUT parameter. + + All arrays returned in OUT variables are allocated by this + function, but are the responsibility of the caller to be freed. + + Parameters: + + IN: comm_handle: handle returned by mpidbg_comm_query() + OUT: comm_num_attrs: length of the array returned in com_attrs; + if 0, the value of com_attrs is undefined. + OUT: comm_attrs: array of length comm_attrs_length containing + keyval/value pairs. + + This function will return: + + MPIDBG_SUCCESS: if the handle is valid, was found, and the OUT + parameters were filled in successfully. + MPIDBG_ERR_NOT_FOUND: if the handle is not valid / found. + MPIDBG_ERR_UNSUPPORTED: if this function is unsupported. +*/ +int mpidbg_comm_query_attrs(struct mpidbg_comm_handle_t *comm_handle, + int *comm_num_attrs, + struct mpidbg_attribute_pair_t *comm_attrs); + +/* Query a handle returned by mpidbg_comm_query() and, if found and + valid, return an array of pending requests on this communicator. + + All arrays returned in OUT variables are allocated by this + function, but are the responsibility of the caller to be freed. + + Parameters: + + IN: comm_handle: handle returned by mpidbg_comm_query() + OUT: comm_num_requests: filled with the length of the + comm_pending_requests array. + OUT: comm_requests: filled with a pointer to an array of pending + requests on this communicator. + + This function will return: + + MPIDBG_SUCCESS: if the handle is valid, was found, and the OUT + parameters were filled in successfully. + MPIDBG_ERR_NOT_FOUND: if the handle is not valid / found. + MPIDBG_ERR_UNSUPPORTED: if this function is unsupported. +*/ +int mpidbg_comm_query_requests(struct mpidbg_comm_handle_t *handle, + int *comm_num_requests, + mqs_taddr_t **comm_pending_requests); + +/* Query a handle returned by mpidbg_comm_query() and, if found and + valid, return arrays of MPI_File and MPI_Win handles that were + derived from this communicator. + + All arrays returned in OUT variables are allocated by this + function, but are the responsibility of the caller to be freed. + + Parameters: + + IN: comm_handle: handle returned by mpidbg_comm_query() + OUT: comm_num_derived_files: filled with the length of the + comm_derived_files array. + OUT: comm_derived_files: filled with a pointer to an array of + MPI_File handles derived from this communicator. + OUT: comm_num_derived_windows: filled with the length of the + comm_derived_windows array. + OUT: comm_derived_windows: filled with a pointer to an array of + MPI_Win handles derived from this communicator. + + This function will return: + + MPIDBG_SUCCESS: if the handle is valid, was found, and the OUT + parameters were filled in successfully. + MPIDBG_ERR_NOT_FOUND: if the handle is not valid / found. + MPIDBG_ERR_UNSUPPORTED: if this function is unsupported. +*/ +int mpidbg_comm_query_derived(struct mpidbg_comm_handle_t *handle, + int *comm_num_derived_files, + mqs_taddr_t **comm_derived_files, + int *comm_num_derived_windows, + mqs_taddr_t **comm_derived_windows); + +/*----------------------------------------------------------------------- + * MPI handle query functions + * MPI_Errhandler + *-----------------------------------------------------------------------*/ + +/* These functions are analogous to the mpidbg_comm_* functions, but + for MPI_Errhandler. */ +int mpidbg_errhandler_query(mqs_image *image, + mqs_image_info *image_info, + mqs_process *process, + mqs_process_info *process_info, + mqs_taddr_t errhandler, + struct mpidbg_errhandler_handle_t **handle); + +/* Free a handle returned by the mpidbg_errhandler_query() function. + + Parameters: + + IN: handle: handle previously returned by mpidbg_errhandler_query() + + This function will return: + + MPIDBG_SUCCESS: if the handle is valid and was freed successfully. + MPIDBG_ERR_NOT_FOUND: if the handle is not valid / found. +*/ +int mpidbg_errhandler_handle_free(struct mpidbg_errhandler_handle_t *handle); + +/* Query a handle returned by mpidbg_comm_query() and, if found and + valid, return basic information about the original communicator. + ***The diff for this file has been truncated for email.*** From padb at googlecode.com Tue Nov 3 16:49:46 2009 From: padb at googlecode.com (padb at googlecode.com) Date: Tue, 03 Nov 2009 16:49:46 +0000 Subject: [padb] r316 committed - Changes to allow the imported header file to compile, I'll have to see... Message-ID: <0016e64b9a1c1e0b3704777a4846@google.com> Revision: 316 Author: apittman Date: Tue Nov 3 08:48:58 2009 Log: Changes to allow the imported header file to compile, I'll have to see about pushing these upstream at some point. http://code.google.com/p/padb/source/detail?r=316 Modified: /branches/mpi3-tools-handles/src/mpihandles_interface.h ======================================= --- /branches/mpi3-tools-handles/src/mpihandles_interface.h Tue Nov 3 08:45:14 2009 +++ /branches/mpi3-tools-handles/src/mpihandles_interface.h Tue Nov 3 08:48:58 2009 @@ -32,7 +32,11 @@ #ifndef __MPIDBG_INTERFACE_H__ #define __MPIDBG_INTERFACE_H__ 1 +/* This isn't going to work... AMP #include "ompi_config.h" +*/ + +#define MPI_MAX_OBJECT_NAME 1024 /* * This file provides interface functions for a debugger to gather @@ -42,9 +46,12 @@ /* Include the Etnus debugger message queue interface so that we can use much of its infrastructure (e.g., the mqs_basic_callbacks, - mqs_image_callbacks, and mqs_process_callbacks). */ + mqs_image_callbacks, and mqs_process_callbacks). + + minfo.c already included this AMP. #define FOR_MPI2 0 #include "msgq_interface.h" +*/ /************************************************************************** * Types and macros From padb at googlecode.com Tue Nov 3 18:18:40 2009 From: padb at googlecode.com (padb at googlecode.com) Date: Tue, 03 Nov 2009 18:18:40 +0000 Subject: [padb] r317 committed - Add a dependency from minfo.c to mpihandle_interface.h. Message-ID: <00504502cb1c100cc704777b8647@google.com> Revision: 317 Author: apittman Date: Tue Nov 3 10:17:28 2009 Log: Add a dependency from minfo.c to mpihandle_interface.h. http://code.google.com/p/padb/source/detail?r=317 Modified: /branches/mpi3-tools-handles/src/Makefile ======================================= --- /branches/mpi3-tools-handles/src/Makefile Wed Oct 28 15:31:42 2009 +++ /branches/mpi3-tools-handles/src/Makefile Tue Nov 3 10:17:28 2009 @@ -7,7 +7,7 @@ FILES = Makefile minfo.c mpi_interface.h padb -minfo.x: minfo.c mpi_interface.h +minfo.x: minfo.c mpi_interface.h mpihandles_interface.h $(CC) minfo.c -o minfo.x -ldl $(CFLAGS) install: minfo.x From padb at googlecode.com Tue Nov 3 19:19:19 2009 From: padb at googlecode.com (padb at googlecode.com) Date: Tue, 03 Nov 2009 19:19:19 +0000 Subject: [padb] r318 committed - Try and load the dbg dll and entry points from the mpi library if it e... Message-ID: <000e0cd14d44f6771104777c5e07@google.com> Revision: 318 Author: apittman Date: Tue Nov 3 11:19:02 2009 Log: Try and load the dbg dll and entry points from the mpi library if it exists. Add a mpi-dbg-dll configuration option to specify which dll to load. Currently doesn't work as the dll provided by OMPI doesn't export the right symbols, also doesn't attempt to do anything once it's opened the DLL. http://code.google.com/p/padb/source/detail?r=318 Modified: /branches/mpi3-tools-handles/src/minfo.c /branches/mpi3-tools-handles/src/padb ======================================= --- /branches/mpi3-tools-handles/src/minfo.c Tue Nov 3 04:15:34 2009 +++ /branches/mpi3-tools-handles/src/minfo.c Tue Nov 3 11:19:02 2009 @@ -13,6 +13,7 @@ #include #include #include "mpi_interface.h" +#include "mpihandles_interface.h" struct dll_entry_points { char *(*dll_error_string) (int); @@ -32,6 +33,90 @@ int (*next_operation) (mqs_process *, mqs_pending_operation *); }; +struct dbg_dll_entry_points { + int (*init_once) (mqs_basic_callbacks *); + int (*interface_version_compatibility) (); + char *(*version_string) (); + int (*dll_taddr_width) (); + int (*init_per_image) (mqs_image *, + mqs_image_callbacks *, + struct mpidbg_handle_info_t *); + void (*finalize_per_image) (mqs_image *, + mqs_image_info *); + int (*init_per_process) (mqs_process *, + mqs_process_callbacks *, + struct mpidbg_handle_info_t *); + void (*finalize_per_process) (mqs_process *, + mqs_process_info *); + int (*comm_query) (mqs_image *, + mqs_image_info *, + mqs_process *, + mqs_process_info *, + mqs_taddr_t, + struct mpidbg_comm_handle_t **); + int (*comm_handle_free) (struct mpidbg_comm_handle_t *); + int (*comm_query_basic) (struct mpidbg_comm_handle_t *, + char comm_name[MPIDBG_MAX_OBJECT_NAME], + enum mpidbg_comm_info_bitmap_t *, + int *, + int *, + int *, + mqs_taddr_t *); + int (*comm_query_procs) (struct mpidbg_comm_handle_t *, + int *, + struct mpidbg_process_t **, + int *, + struct mpidbg_process_t **); + int (*comm_query_topo) (struct mpidbg_comm_handle_t *, + int *, + int **, + int **); + int (*comm_query_attrs) (struct mpidbg_comm_handle_t *, + int *, + struct mpidbg_attribute_pair_t *); + int (*comm_query_requests) (struct mpidbg_comm_handle_t *, + int *, + mqs_taddr_t **); + int (*comm_query_derived) (struct mpidbg_comm_handle_t *, + int *, + mqs_taddr_t **, + int *, + mqs_taddr_t **); + int (*errhandler_query) (mqs_image *, + mqs_image_info *, + mqs_process *, + mqs_process_info *, + mqs_taddr_t, + struct mpidbg_errhandler_handle_t **); + int (*errhandler_handle_free) (struct mpidbg_errhandler_handle_t *); + int (*errhandler_query_basic) (struct mpidbg_errhandler_handle_t *, + char errhandler_name[MPIDBG_MAX_OBJECT_NAME], + enum mpidbg_errhandler_info_bitmap_t *, + int *, + mqs_taddr_t *); + int (*errhandler_query_handles) (struct mpidbg_errhandler_handle_t *, + int *, + mqs_taddr_t **); + int (*errhandler_query_callback) (struct mpidbg_errhandler_handle_t *, + mqs_taddr_t *); + int (*request_query) (mqs_image *, + mqs_image_info *, + mqs_process *, + mqs_process_info *, + mqs_taddr_t, + struct mpidbg_request_handle_t **); + int (*request_handle_free) (struct mpidbg_request_handle_t *); + int (*status_query) (mqs_image *, + mqs_image_info *, + mqs_process *, + mqs_process_info *, + mqs_taddr_t, + struct mpidbg_status_handle_t **); + int (*status_handle_free) (struct mpidbg_status_handle_t *); +}; + +struct dbg_dll_entry_points dbg_dll_ep = {}; + struct image { mqs_image_info *blob; }; @@ -511,11 +596,11 @@ load_ops(target_process,mqs_pending_sends); } -#define DLSYM_LAX(VAR,HANDLE,NAME) VAR.NAME = dlsym(HANDLE,"mqs_" #NAME) - -#define DLSYM(VAR,HANDLE,NAME) do { \ - if ( (DLSYM_LAX(VAR,HANDLE,NAME)) == NULL ) { \ - show_warning("Failed to load symbol mqs_" #NAME); \ +#define DLSYM_LAX(VAR,HANDLE,PREFIX,NAME) VAR.NAME = dlsym(HANDLE,#PREFIX "_" #NAME) + +#define DLSYM(VAR,HANDLE,PREFIX,NAME) do { \ + if ( (DLSYM_LAX(VAR,HANDLE,PREFIX,NAME)) == NULL ) { \ + show_warning("Failed to load symbol " #PREFIX "_" #NAME); \ return -1; \ } \ } while (0) @@ -523,35 +608,84 @@ /* Try and load the dll from a given filename, returns true if successfull. * populates the contents of dll_ep if true. */ -int load_msgq_dll(char *filename) -{ - void *dlhandle; - - dlhandle = dlopen(filename,RTLD_NOW); - if ( ! dlhandle ) - return -1; - - DLSYM(dll_ep,dlhandle,setup_basic_callbacks); - DLSYM(dll_ep,dlhandle,setup_image); - DLSYM(dll_ep,dlhandle,image_has_queues); - DLSYM(dll_ep,dlhandle,setup_process); - DLSYM(dll_ep,dlhandle,process_has_queues); - DLSYM(dll_ep,dlhandle,dll_error_string); - DLSYM(dll_ep,dlhandle,update_communicator_list); - DLSYM(dll_ep,dlhandle,setup_communicator_iterator); - DLSYM(dll_ep,dlhandle,get_communicator); - DLSYM(dll_ep,dlhandle,next_communicator); - DLSYM(dll_ep,dlhandle,setup_operation_iterator); - DLSYM(dll_ep,dlhandle,next_operation); +int load_msgq_dll(void *dlhandle) +{ + DLSYM(dll_ep,dlhandle,mqs,setup_basic_callbacks); + DLSYM(dll_ep,dlhandle,mqs,setup_image); + DLSYM(dll_ep,dlhandle,mqs,image_has_queues); + DLSYM(dll_ep,dlhandle,mqs,setup_process); + DLSYM(dll_ep,dlhandle,mqs,process_has_queues); + DLSYM(dll_ep,dlhandle,mqs,dll_error_string); + DLSYM(dll_ep,dlhandle,mqs,update_communicator_list); + DLSYM(dll_ep,dlhandle,mqs,setup_communicator_iterator); + DLSYM(dll_ep,dlhandle,mqs,get_communicator); + DLSYM(dll_ep,dlhandle,mqs,next_communicator); + DLSYM(dll_ep,dlhandle,mqs,setup_operation_iterator); + DLSYM(dll_ep,dlhandle,mqs,next_operation); /* "Optional" symbols, these may or may not occour in the dll, if they * are we can make use of them but if they aren't then that's fine too */ - DLSYM_LAX(dll_ep,dlhandle,get_comm_group); - DLSYM_LAX(dll_ep,dlhandle,get_global_rank); - DLSYM_LAX(dll_ep,dlhandle,get_comm_coll_state); + DLSYM_LAX(dll_ep,dlhandle,mqs,get_comm_group); + DLSYM_LAX(dll_ep,dlhandle,mqs,get_global_rank); + DLSYM_LAX(dll_ep,dlhandle,mqs,get_comm_coll_state); return 0; } + +int load_dbg_dll(void *dlhandle) +{ + DLSYM(dbg_dll_ep,dlhandle,mpidbg,init_once); + DLSYM(dbg_dll_ep,dlhandle,mpidbg,version_string); + DLSYM(dbg_dll_ep,dlhandle,mpidbg,dll_taddr_width); + DLSYM(dbg_dll_ep,dlhandle,mpidbg,init_per_image); + DLSYM(dbg_dll_ep,dlhandle,mpidbg,finalize_per_image); + DLSYM(dbg_dll_ep,dlhandle,mpidbg,init_per_process); + DLSYM(dbg_dll_ep,dlhandle,mpidbg,finalize_per_process); + DLSYM(dbg_dll_ep,dlhandle,mpidbg,comm_query); + DLSYM(dbg_dll_ep,dlhandle,mpidbg,comm_handle_free); + DLSYM(dbg_dll_ep,dlhandle,mpidbg,comm_query_basic); + DLSYM(dbg_dll_ep,dlhandle,mpidbg,comm_query_procs); + DLSYM(dbg_dll_ep,dlhandle,mpidbg,comm_query_topo); + DLSYM(dbg_dll_ep,dlhandle,mpidbg,comm_query_attrs); + DLSYM(dbg_dll_ep,dlhandle,mpidbg,comm_query_requests); + DLSYM(dbg_dll_ep,dlhandle,mpidbg,comm_query_derived); + DLSYM(dbg_dll_ep,dlhandle,mpidbg,errhandler_query); + DLSYM(dbg_dll_ep,dlhandle,mpidbg,errhandler_handle_free); + DLSYM(dbg_dll_ep,dlhandle,mpidbg,errhandler_query_basic); + DLSYM(dbg_dll_ep,dlhandle,mpidbg,errhandler_query_handles); + DLSYM(dbg_dll_ep,dlhandle,mpidbg,errhandler_query_callback); + DLSYM(dbg_dll_ep,dlhandle,mpidbg,request_query); + DLSYM(dbg_dll_ep,dlhandle,mpidbg,request_handle_free); + DLSYM(dbg_dll_ep,dlhandle,mpidbg,status_query); + DLSYM(dbg_dll_ep,dlhandle,mpidbg,status_handle_free); + + return 0; +} + +int load_dll (char *dll_name, int (*load_fn)(void *)) +{ + void *dlhandle; + + printf("Trying %s\n",dll_name); + dlhandle = dlopen(dll_name,RTLD_NOW); + if ( dlhandle == NULL ) { + show_warning(dlerror()); + + dlhandle = dlopen(dll_name,RTLD_LAZY); + if ( dlhandle == NULL ) { + show_warning(dlerror()); + return -1; + } + } + + if ( load_fn(dlhandle) == 0 ) { + free(dll_name); + return mqs_ok; + } + dlclose(dlhandle); + + return -1; +} #define PATH_MAX 1024 @@ -559,10 +693,11 @@ * trying each one in turn. Return 0 if and when we managed to load one, * -1 otherwise */ -int find_and_load_dll_from_loc_array() { +int find_and_load_dll_from_loc_array(char *sym_name, int (*load_fn)(void *)) +{ void **remote_array; char *dll_name; - void *locations = find_sym("sym","mpimsgq_dll_locations"); + void *locations = find_sym("sym",sym_name); if ( locations == NULL ) return -1; @@ -589,8 +724,7 @@ goto error_out; } else { - if ( load_msgq_dll(dll_name) == 0 ) { - free(dll_name); + if ( load_dll(dll_name,load_fn) == mqs_ok ) { return mqs_ok; } } @@ -608,14 +742,14 @@ dll_name = getenv("MPINFO_DLL"); if ( dll_name != NULL ) { - if ( load_msgq_dll(dll_name) != 0 ) { + if ( load_dll(dll_name,load_msgq_dll) != mqs_ok ) { die("Could not load symbols from dll"); } return; } /* Try the new (proposed) dll specification mechanism */ - if ( find_and_load_dll_from_loc_array() == mqs_ok ) + if ( find_and_load_dll_from_loc_array("mpimsgq_dll_locations",load_msgq_dll) == mqs_ok ) return; void *base = find_sym("sym","MPIR_dll_name"); @@ -626,10 +760,22 @@ if ( fetch_string(NULL,dll_name,(mqs_taddr_t)base,PATH_MAX) != 0 ) { die("Could not read value of MPIR_dll_name"); } - if ( load_msgq_dll(dll_name) != 0 ) { + if ( load_dll(dll_name,load_msgq_dll) != mqs_ok ) { die("Could not load symbols from dll"); } } + +int find_and_load_dbg_dll () +{ + char *dll_name = getenv("MPI_DBG_DLL"); + if ( dll_name != NULL ) { + if ( load_dll(dll_name,load_dbg_dll) != mqs_ok ) { + show_warning("Could not load symbols from dbg dll"); + return -1; + } + } + return find_and_load_dll_from_loc_array("mpidbg_dll_locations",load_dbg_dll); +} int main () @@ -644,6 +790,11 @@ mqs_process *target_process = (mqs_process *)&process; find_and_load_dll(); + + /* Try the new (proposed) dll specification mechanism */ + if ( find_and_load_dbg_dll() == mqs_ok ) { + printf("Loaded the other dll as well\n"); + } assert(dll_ep.setup_basic_callbacks != NULL); ======================================= --- /branches/mpi3-tools-handles/src/padb Mon Nov 2 09:55:58 2009 +++ /branches/mpi3-tools-handles/src/padb Tue Nov 3 11:19:02 2009 @@ -6064,6 +6064,10 @@ 'Process does not appear to be using MPI (No MPIR_dll_name symbol)' ); } + + if ( defined $carg->{mpi_dbg_dll} ) { + $ENV{MPI_DBG_DLL} = $carg->{mpi_dbg_dll}; + } if ( defined $carg->{mpi_dll} ) { $ENV{MPINFO_DLL} = $carg->{mpi_dll}; @@ -6091,6 +6095,10 @@ 'Process does not appear to be using MPI (No MPIR_dll_name symbol)' ); } + + if ( defined $carg->{mpi_dbg_dll} ) { + $ENV{MPI_DBG_DLL} = $carg->{mpi_dbg_dll}; + } if ( defined $carg->{mpi_dll} ) { $ENV{MPINFO_DLL} = $carg->{mpi_dll}; @@ -8628,8 +8636,10 @@ arg_short => 'q', handler => \&show_queue, help => 'Show the message queues', - options_i => { mpi_dll => undef, } - + options_i => { + mpi_dll => undef, + mpi_dbg_dll => undef, + }, }; $allfns{kill} = { @@ -8651,7 +8661,10 @@ arg_long => 'mpi-queue', arg_short => 'Q', help => 'Show MPI message queues', - options_i => { mpi_dll => undef, } + options_i => { + mpi_dll => undef, + mpi_dbg_dll => undef, + }, }; $allfns{deadlock} = { @@ -8660,12 +8673,14 @@ arg_short => 'j', help => 'Run deadlock detection algorithm', out_handler => \&mpi_deadlock_detect, - options_i => { mpi_dll => undef, }, options_bool => { show_group_members => 'no', show_all_groups => 'no', }, - + options_i => { + mpi_dll => undef, + mpi_dbg_dll => undef, + }, }; $allfns{pinfo} = { @@ -8677,8 +8692,7 @@ proc_shows_fds => 'no', proc_shows_maps => 'no', proc_shows_stat => 'no', - } - + }, }; $allfns{proc_summary} = { @@ -8698,7 +8712,6 @@ proc_shows_maps => 'no', proc_show_header => 'yes', reverse_sort_order => 'no', - }, secondary => [ { @@ -8738,8 +8751,7 @@ type => '!', default => 1, }, - ] - + ] }; $allfns{stack_long} = { @@ -8757,8 +8769,9 @@ out_handler => \&show_mpi_watch, options_i => { mpi_dll => undef, - mpi_watch_file => undef - } + mpi_dbg_dll => undef, + mpi_watch_file => undef, + }, }; $allfns{lstopo} = { @@ -8804,14 +8817,12 @@ options_i => { stats_name => undef, stats_sort_key => 'vp', - }, options_bool => { stats_reverse => 'no', stats_short => 'no', show_all_stats => 'no', - } - + } }; $allfns{qsnet_groups} = { handler_all => \&inner_show_stats, From thipadin.seng-long at bull.net Wed Nov 4 09:50:19 2009 From: thipadin.seng-long at bull.net (thipadin.seng-long at bull.net) Date: Wed, 4 Nov 2009 10:50:19 +0100 Subject: [padb] =?iso-8859-1?q?R=E9f=2E_=3A_Re=3A_=5Bpadb-devel=5D_Patch_f?= =?iso-8859-1?q?or_Support_of_PBS__Pro_resource_manager?= Message-ID: Hi, Thanks you for your response. So I'll try to adapt my written from the last release, but I wonder from which one as patches go new every day. So can you commit a branch (or something) where I can start from, it would be a great help. Thanks. Thipadin. Ashley Pittman 10/30/2009 07:26 PM Pour : thipadin.seng-long at bull.net cc : padb-devel at pittman.org.uk, florence.vallee at bull.net, francois.wellenreiter at bull.net, Sylvain.JEAUGEY at bull.net Objet : Re: [padb-devel] Patch for Support of PBS Pro resource manager On Fri, 2009-10-30 at 17:06 +0100, thipadin.seng-long at bull.net wrote: > I'm a Bull software engineer and been working for a while on padb to > adapt it to PBS Pro environment. > The modified padb has been working fine in our MPI_Bull2 / Pbs Pro > environment. > So I am sending you the patch for you to integrate in the common padb > branch. > I have started from padb 2.5 version, sorry. > If you want to discuss about my coding I am ready, feel free to > comment before you commit it. > Many other resources managers (slurm-orte, lsf-mpd, lsf-orte) will > follow. First off that's great! I'm always keen to see padb better and available to a wider range of people. Secondly as you'll see if you look at the history I've done a lot of work since the 2.5 release so several aspects of the design are different. For the most part this should make things simpler to code for however it does mean changes to the code you have written. I'm happy to integrate patches which add new resource managers as the danger of breaking anything else is low, I'd like to see if we can get away without they lpbs_get_script_name() function though. Are you able to upgrade to the HEAD code and submit patches based on that? have a look at @inner_conf and in particular slurm_job_step for how to pass on $server and $port to the inner processes. Ashley, -- Ashley Pittman, Bath, UK. Padb - A parallel job inspection tool for cluster computing http://padb.pittman.org.uk -------------- next part -------------- An HTML attachment was scrubbed... URL: From ashley at pittman.co.uk Wed Nov 4 10:34:43 2009 From: ashley at pittman.co.uk (Ashley Pittman) Date: Wed, 04 Nov 2009 10:34:43 +0000 Subject: [padb] =?iso-8859-1?q?R=E9f=2E_=3A_Re=3A_=5Bpadb-devel=5D_Patch_f?= =?iso-8859-1?q?or_Support_of_PBS_Pro_resource_manager?= In-Reply-To: References: Message-ID: <1257330883.4444.8.camel@alpha> Thipadin. Most of my recent commits have been to do with gdb interactions and getting Solaris support up and running so should not interfere with your work, the more intrusive stuff I'm doing with library callbacks I created a branch for yesterday. The exception to this is r311 which you probably want to look at, the new "mpirun" resource manager and your code both call pdsh so I abstracted this logic out into the infrastructure so you can re-use it. I'm happy to create a branch for you if you want to something more stable to work with, you'll have to provide me with a google login if I'm to give you commit access to it. I also created a 2.5 branch on friday which I should have done when it was released, I believe this is what your original work was based against so this should help you manage the code you have. Ashley, On Wed, 2009-11-04 at 10:50 +0100, thipadin.seng-long at bull.net wrote: > > Hi, > Thanks you for your response. So I'll try to adapt my written from the > last release, but I wonder from which one as patches go new every > day. > So can you commit a branch (or something) where I can start from, it > would be a great help. > Thanks. Thipadin. -- Ashley Pittman, Bath, UK. Padb - A parallel job inspection tool for cluster computing http://padb.pittman.org.uk From thipadin.seng-long at bull.net Wed Nov 4 12:31:02 2009 From: thipadin.seng-long at bull.net (thipadin.seng-long at bull.net) Date: Wed, 4 Nov 2009 13:31:02 +0100 Subject: [padb] =?iso-8859-1?q?R=E9f=2E_=3A_Re=3A_R=E9f=2E_=3A_Re=3A_=5Bpa?= =?iso-8859-1?q?db-devel=5D_Patch__for_Support_of_PBS_Pro_resource_manager?= Message-ID: As I can get it, you propose me two things: 1-start from r311 2-create branch with stable state for a while (must provide a google login)and merge branch later. So what's the best thing for you ? For me I don't need access to commit, just check out is enough. And I don't want to hold parallel version any longer, just want to get to common version quickly. Thipadin. Ashley Pittman 11/04/2009 11:34 AM Pour : thipadin.seng-long at bull.net cc : florence.vallee at bull.net, francois.wellenreiter at bull.net, padb-devel at pittman.org.uk, Sylvain.JEAUGEY at bull.net Objet : Re: R?f. : Re: [padb-devel] Patch for Support of PBS Pro resource manager Thipadin. Most of my recent commits have been to do with gdb interactions and getting Solaris support up and running so should not interfere with your work, the more intrusive stuff I'm doing with library callbacks I created a branch for yesterday. The exception to this is r311 which you probably want to look at, the new "mpirun" resource manager and your code both call pdsh so I abstracted this logic out into the infrastructure so you can re-use it. I'm happy to create a branch for you if you want to something more stable to work with, you'll have to provide me with a google login if I'm to give you commit access to it. I also created a 2.5 branch on friday which I should have done when it was released, I believe this is what your original work was based against so this should help you manage the code you have. Ashley, On Wed, 2009-11-04 at 10:50 +0100, thipadin.seng-long at bull.net wrote: > > Hi, > Thanks you for your response. So I'll try to adapt my written from the > last release, but I wonder from which one as patches go new every > day. > So can you commit a branch (or something) where I can start from, it > would be a great help. > Thanks. Thipadin. -- Ashley Pittman, Bath, UK. Padb - A parallel job inspection tool for cluster computing http://padb.pittman.org.uk -------------- next part -------------- An HTML attachment was scrubbed... URL: From ashley at pittman.co.uk Wed Nov 4 13:01:57 2009 From: ashley at pittman.co.uk (Ashley Pittman) Date: Wed, 04 Nov 2009 13:01:57 +0000 Subject: [padb] =?iso-8859-1?q?R=E9f=2E_=3A_Re=3A_R=E9f=2E_=3A_Re=3A_=5Bpa?= =?iso-8859-1?q?db-devel=5D_Patch_for_Support_of_PBS_Pro_resource_manager?= In-Reply-To: References: Message-ID: <1257339717.4444.50.camel@alpha> On Wed, 2009-11-04 at 13:31 +0100, thipadin.seng-long at bull.net wrote: > > As I can get it, you propose me two things: > 1-start from r311 > 2-create branch with stable state for a while (must provide a google > login)and merge branch later. > So what's the best thing for you ? For me I don't need access to > commit, just check out is enough. > And I don't want to hold parallel version any longer, just want to get > to common version quickly. You could either work against the head which shouldn't be getting commits which would conflict with your work or you could work against r311, I'll be happy with either. You can checkout r311 using the following command if you need to. $ svn co -r 311 https://padb.googlecode.com/svn/trunk padb Looking over your patch it should all be straightforward other than the server part, does PBS often have two active servers on the same machine? What might a list of active jobs look like in this case? Ashley -- Ashley Pittman, Bath, UK. Padb - A parallel job inspection tool for cluster computing http://padb.pittman.org.uk From padb at googlecode.com Wed Nov 4 13:08:01 2009 From: padb at googlecode.com (padb at googlecode.com) Date: Wed, 04 Nov 2009 13:08:01 +0000 Subject: [padb] r319 committed - Avoid the use of < and > characters in stack trace variable display as... Message-ID: <0016e64b0bc0ef55d404778b4c79@google.com> Revision: 319 Author: apittman Date: Wed Nov 4 05:07:25 2009 Log: Avoid the use of < and > characters in stack trace variable display as it plays badly when embedding the output in html for displaying on the web. Pass all code that needs this formatting through a pretify_variable function which for now just returns the string it's passed, maybe in the future this will be able to do something better. http://code.google.com/p/padb/source/detail?r=319 Modified: /trunk/src/padb ======================================= --- /trunk/src/padb Mon Nov 2 09:55:58 2009 +++ /trunk/src/padb Wed Nov 4 05:07:25 2009 @@ -3423,6 +3423,18 @@ return; } + +# Controls how to display "expanded" variables in stack traces, that is +# where the value display isn't the contents of the variable but some +# mutation of that, either the name of a MPI Communicator, or a pointer +# description or that it's value is too long. +# For text based output <$var> works well IMHO however that isn't valid +# html, for now this just returns $var but perhaps having a +# --format-for-html option would be the way to go in future. +sub pretify_variable { + my $var = shift; + return $var; +} sub _add_data_to_point_on_tree { my ( $tree, $d, $l, $max_show, $peer, @vars ) = @_; @@ -3450,8 +3462,8 @@ . rng_convert_to_user( $d->{target_data}{$key}{ $values[0] } ); push @{ $tree->{$peer}->{aux} }, $line; } elsif ( @values > $max_show ) { - my $line = - " $vref->{type_name}: "; + my $line = " $vref->{type_name}: " + . pretify_variable("more than $max_show distinct values"); push @{ $tree->{$peer}->{aux} }, $line; } else { push @{ $tree->{$peer}->{aux} }, " $vref->{type_name}:"; @@ -6494,19 +6506,20 @@ my $pval = _hex($ptr); if ( $ptr eq '0x0' ) { - return ''; + return pretify_variable('null pointer'); } foreach my $rgn ( @{ $gdb->{maps} } ) { if ( $pval >= $rgn->{start} and $pval <= $rgn->{end} ) { if ( defined $rgn->{file} ) { - return "{perm} ($rgn->{file})>"; + return pretify_variable( + "valid pointer perm=$rgn->{perm} ($rgn->{file})"); } else { - return "{perm}>"; + return pretify_variable("valid pointer perm=$rgn->{perm}"); } } } - return "<$ptr (Invalid pointer)>"; + return pretify_variable("$ptr (Invalid pointer)"); } sub expand_var_hash { @@ -6565,7 +6578,7 @@ # If you try and read a value which claims to be optimised away it # will return a value of zero, hard to know how to handle this but # not reporting it is probably the better of the two options. - return + return pretify_variable('value optimized out') if ( defined $arg->{value} and $arg->{value} eq '' ); # Char * pointers are already correctly handled by gdb with set @@ -6589,7 +6602,7 @@ my $expanded_var = custom_expand_var( $gdb, $type, $arg->{name}, $addr ); if ( defined $expanded_var ) { - return "<$expanded_var>"; + return pretify_variable($expanded_var); } } @@ -6607,7 +6620,7 @@ } if ( $pointer eq '0x0' ) { - return ''; + return pretify_variable('null pointer'); } # Try and print the contents of the pointer, this works fine @@ -7422,7 +7435,8 @@ target_key_pair( $vp, $fl . '|var|' . $loc->{name}, - '' + pretify_variable( + 'value too long to display') ); } else { target_key_pair( $vp, @@ -7446,7 +7460,8 @@ target_key_pair( $vp, $fl . '|var|' . $par->{name}, - '' + pretify_variable( + 'value too long to display') ); } else { target_key_pair( $vp, From thipadin.seng-long at bull.net Wed Nov 4 13:09:53 2009 From: thipadin.seng-long at bull.net (thipadin.seng-long at bull.net) Date: Wed, 4 Nov 2009 14:09:53 +0100 Subject: [padb] =?iso-8859-1?q?R=E9f=2E_=3A_Re=3A_R=E9f=2E_=3A_Re=3A_R=E9f?= =?iso-8859-1?q?=2E_=3A_Re=3A_=5B_padb-devel=5D_Patchfor_Support_of_PBS_Pr?= =?iso-8859-1?q?o_resource_manager?= Message-ID: So let go with r311. Thanks. Thipadin. Ashley Pittman 11/04/2009 02:01 PM Pour : thipadin.seng-long at bull.net cc : florence.vallee at bull.net, francois.wellenreiter at bull.net, padb-devel at pittman.org.uk, Sylvain.JEAUGEY at bull.net Objet : Re: R?f. : Re: R?f. : Re: [padb-devel] Patch for Support of PBS Pro resource manager On Wed, 2009-11-04 at 13:31 +0100, thipadin.seng-long at bull.net wrote: > > As I can get it, you propose me two things: > 1-start from r311 > 2-create branch with stable state for a while (must provide a google > login)and merge branch later. > So what's the best thing for you ? For me I don't need access to > commit, just check out is enough. > And I don't want to hold parallel version any longer, just want to get > to common version quickly. You could either work against the head which shouldn't be getting commits which would conflict with your work or you could work against r311, I'll be happy with either. You can checkout r311 using the following command if you need to. $ svn co -r 311 https://padb.googlecode.com/svn/trunk padb Looking over your patch it should all be straightforward other than the server part, does PBS often have two active servers on the same machine? What might a list of active jobs look like in this case? Ashley -- Ashley Pittman, Bath, UK. Padb - A parallel job inspection tool for cluster computing http://padb.pittman.org.uk -------------- next part -------------- An HTML attachment was scrubbed... URL: From ashley at pittman.co.uk Thu Nov 5 16:53:49 2009 From: ashley at pittman.co.uk (Ashley Pittman) Date: Thu, 05 Nov 2009 16:53:49 +0000 Subject: [padb] [padb-devel] Simple Makefile patch In-Reply-To: <1257199612.25087.54.camel@alpha> References: <20091102163826.GC53330@sun.com> <1257181765.3430.334.camel@alpha> <20091102172426.GD53330@sun.com> <1257184825.3430.337.camel@alpha> <20091102184534.GE53330@sun.com> <1257188512.3430.343.camel@alpha> <20091102191228.GF53330@sun.com> <1257190832.3430.379.camel@alpha> <20091102195827.GG53330@sun.com> <1257193387.25087.24.camel@alpha> <20091102213052.GH53330@sun.com> <1257199612.25087.54.camel@alpha> Message-ID: <1257440029.3633.20.camel@alpha> Ethan, I had a thought about this last night, it could be that the scanf qualifier %p isn't working, googling around for some docs suggest that % p won't work, can you try the attached patch to see if that fixes the issue? Failing that we might have to wait until I can next get my hands on a sun machine. http://docs.sun.com/app/docs/doc/816-5168/printf-3ucb?a=view Ashley, -- Ashley Pittman, Bath, UK. Padb - A parallel job inspection tool for cluster computing http://padb.pittman.org.uk -------------- next part -------------- A non-text attachment was scrubbed... Name: padb-scanf.patch Type: text/x-patch Size: 1168 bytes Desc: not available URL: From ethan.mallove at sun.com Thu Nov 5 18:20:33 2009 From: ethan.mallove at sun.com (Ethan Mallove) Date: Thu, 5 Nov 2009 13:20:33 -0500 Subject: [padb] [padb-devel] Simple Makefile patch In-Reply-To: <1257199612.25087.54.camel@alpha> References: <20091102172426.GD53330@sun.com> <1257184825.3430.337.camel@alpha> <20091102184534.GE53330@sun.com> <1257188512.3430.343.camel@alpha> <20091102191228.GF53330@sun.com> <1257190832.3430.379.camel@alpha> <20091102195827.GG53330@sun.com> <1257193387.25087.24.camel@alpha> <20091102213052.GH53330@sun.com> <1257199612.25087.54.camel@alpha> Message-ID: <20091105182033.GL53330@sun.com> On Mon, Nov/02/2009 10:06:52PM, Ashley Pittman wrote: > On Mon, 2009-11-02 at 16:30 -0500, Ethan Mallove wrote: > > > > Could you send me the log file for a run with that option set please. > > > > The log files are the same: > > > > $ cat /tmp/padb-minfo-debug-log-0-IkyIH9 > > req: sym MPIR_dll_name > > ok 0xff1f9824 > > zzz: str:35 dmsg > > Could not find MPIR_dll_name symbol > > zzz: str:3 exit > > die > > That would imply that it's found it but then not recognised it. > > Could you add some printfs to the find_sym() and ask() functions to see > what's going on, it's the very first contact so it's likely that one of > the core functions is failing, perhaps the read(0 is a problem. The > attached patch should tell us more. > I don't think any of the printf's fired: $ padb --debug=all --config-option rmgr=mpirun --full-report=15961 DEBUG (config): 0: Finished setting configuration options padb version 3.n (Revision 312) full job report for job 15961 DEBUG (pcmd): 1: Loaded pcmd data DEBUG (verbose): 1: There are 1 processes over 1 hosts DEBUG (verbose): 1: Remote process data available on frontend DEBUG (show_cmd): 1: /home/em162155/software/SunOS/sparc/padb/bin/padb --inner DEBUG (signon): 2: Received last signon, connecting to inner DEBUG (ctree): 2: connection tree DEBUG (full_duplex): 2: Sending command to inner, 364 bytes DEBUG (full_duplex): 2: Reply from inner, 316 bytes DEBUG (full_duplex): 2: Sending command to inner, 64 bytes DEBUG (full_duplex): 3: Reply from inner, 384 bytes DEBUG (full_duplex): 3: Sending command to inner, 36 bytes DEBUG (tdata): 3: Target data Namespace: "ERROR" Error message from /home/em162155/software/SunOS/sparc/padb/bin/minfo.x: Could not find MPIR_dll_name symbol [0] Namespace: "FOUND" yes [0] Warning: errors reported by some ranks ======== [0]: Error message from /home/em162155/software/SunOS/sparc/padb/bin/minfo.x: Could not find MPIR_dll_name symbol ======== DEBUG (full_duplex): 5: Reply from inner, 432 bytes DEBUG (full_duplex): 5: Sending command to inner, 472 bytes DEBUG (tdata): 5: Target data Namespace: "ERROR" Error message from /home/em162155/software/SunOS/sparc/padb/bin/minfo.x: Could not find MPIR_dll_name symbol [0] Namespace: "FOUND" yes [0] Warning: errors reported by some ranks ======== [0]: Error message from /home/em162155/software/SunOS/sparc/padb/bin/minfo.x: Could not find MPIR_dll_name symbol ======== Total: 0 communicators, no communication data recorded. DEBUG (full_duplex): 6: Reply from inner, 1380 bytes DEBUG (full_duplex): 6: Sending command to inner, 28 bytes DEBUG (tdata): 6: Target data Namespace: ",main() at hello_c.c:18|var|argc" 1 [0] Namespace: ",main() at hello_c.c:18|var|argv" 0xffbfe304 [0] Namespace: ",main() at hello_c.c:18|var|rank" 0 [0] Namespace: ",main() at hello_c.c:18|var|size" -4201424 [0] Namespace: "FOUND" yes [0] Namespace: "main() at hello_c.c:18|locals" rank,size [0] Namespace: "main() at hello_c.c:18|params" argc,argv [0] Namespace: "main() at hello_c.c:18|var_type|argc" int [0] Namespace: "main() at hello_c.c:18|var_type|argv" char ** [0] Namespace: "main() at hello_c.c:18|var_type|rank" int [0] Namespace: "main() at hello_c.c:18|var_type|size" int [0] DEBUG (tree): 6: Making the tree DEBUG (tree): 6: Enhancing the tree DEBUG (tree): 6: Formatting the tree DEBUG (tree): 6: Displaying the tree ----------------- [0] (1 processes) ----------------- main() at hello_c.c:18 params int argc = '1' [0] char ** argv = '0xffbfe304' [0] locals int rank = '0' [0] int size = '-4201424' [0] ----------------- [0] (1 processes) ----------------- sleep() at ?:? ___nanosleep() at ?:? DEBUG (tree): 6: Done DEBUG (full_duplex): 6: Reply from inner, 84 bytes DEBUG (verbose): 6: Completed command em162155 $ > I assume it compiles without warnings? Right, it compiles w/o warnings. -Ethan > > Ashley, > > -- > > Ashley Pittman, Bath, UK. > > Padb - A parallel job inspection tool for cluster computing > http://padb.pittman.org.uk > Index: minfo.c > =================================================================== > --- minfo.c (revision 310) > +++ minfo.c (working copy) > @@ -117,10 +117,14 @@ > printf("req: %s\n",req); > fflush(NULL); > nbytes = read(0,buff,QUERY_SIZE+3); > - if ( nbytes < 0 ) > + if ( nbytes < 0 ) { > + printf("Read returned %d\n",nbytes); > return -1; > - if ( memcmp(buff,"ok ",3 ) ) > + } > + if ( memcmp(buff,"ok ",3 ) ) { > + show_warning("ask request got nack\n"); > return -1; > + } > buff[nbytes-1] = '\000'; > memcpy(ans,&buff[3],nbytes -3); > return 0; > @@ -134,8 +138,10 @@ > sprintf(req,"%s %s",type,name); > > i = ask(req,ans); > - if ( i != 0 ) > + if ( i != 0 ) { > + show_warning("ask failed\n"); > return NULL; > + } > > i = sscanf(ans, "%p",&addr); > if ( i != 1 ) { > @@ -533,8 +539,8 @@ > DLSYM(dll_ep,dlhandle,next_communicator); > DLSYM(dll_ep,dlhandle,setup_operation_iterator); > DLSYM(dll_ep,dlhandle,next_operation); > - DLSYM(dll_ep,dlhandle,get_comm_group); > > + DLSYM_LAX(dll_ep,dlhandle,get_comm_group); > DLSYM_LAX(dll_ep,dlhandle,get_global_rank); > DLSYM_LAX(dll_ep,dlhandle,get_comm_coll_state); > return 0; From ethan.mallove at sun.com Thu Nov 5 18:23:02 2009 From: ethan.mallove at sun.com (Ethan Mallove) Date: Thu, 5 Nov 2009 13:23:02 -0500 Subject: [padb] [padb-devel] Simple Makefile patch In-Reply-To: <1257440029.3633.20.camel@alpha> References: <1257184825.3430.337.camel@alpha> <20091102184534.GE53330@sun.com> <1257188512.3430.343.camel@alpha> <20091102191228.GF53330@sun.com> <1257190832.3430.379.camel@alpha> <20091102195827.GG53330@sun.com> <1257193387.25087.24.camel@alpha> <20091102213052.GH53330@sun.com> <1257199612.25087.54.camel@alpha> <1257440029.3633.20.camel@alpha> Message-ID: <20091105182302.GM53330@sun.com> On Thu, Nov/05/2009 04:53:49PM, Ashley Pittman wrote: > > Ethan, > > I had a thought about this last night, it could be that the scanf > qualifier %p isn't working, googling around for some docs suggest that % > p won't work, can you try the attached patch to see if that fixes the > issue? Failing that we might have to wait until I can next get my hands > on a sun machine. > > http://docs.sun.com/app/docs/doc/816-5168/printf-3ucb?a=view gcc minfo.c -o minfo.x -ldl -Wall -g minfo.c: In function 'find_sym': minfo.c:146: warning: format '%lx' expects type 'long unsigned int *', but argument 3 has type 'void **' minfo.c:148: warning: format '%lx' expects type 'long unsigned int', but argument 3 has type 'void *' minfo.c: In function '_find_data': minfo.c:306: warning: format '%lx' expects type 'long unsigned int', but argument 3 has type 'void *' minfo.c: In function 'fetch_string': minfo.c:362: warning: format '%lx' expects type 'long unsigned int', but argument 4 has type 'void *' minfo.c: In function 'show_op': minfo.c:468: warning: format '%lx' expects type 'long unsigned int', but argument 3 has type 'void *' -Ethan > > Ashley, > > -- > > Ashley Pittman, Bath, UK. > > Padb - A parallel job inspection tool for cluster computing > http://padb.pittman.org.uk > Index: minfo.c > =================================================================== > --- minfo.c (revision 319) > +++ minfo.c (working copy) > @@ -141,9 +141,9 @@ > if ( i != 0 ) > return NULL; > > - i = sscanf(ans, "%p",&addr); > + i = sscanf(ans, "%lx",&addr); > if ( i != 1 ) { > - printf("Failed sscanf %d %p\n",i,addr); > + printf("Failed sscanf %d %lx\n",i,addr); > return NULL; > } > > @@ -301,7 +301,7 @@ > if ( size == 0 ) > return mqs_ok; > > - sprintf(req,"data %p %d",(void *)addr,size); > + sprintf(req,"data 0x%lx %d",(void *)addr,size); > > i = ask(req,ans); > if ( i != 0 ) > @@ -360,7 +360,7 @@ > char *ans = malloc(size+16); > int i; > > - sprintf(req,"string %d %p",size,(void *)remote); > + sprintf(req,"string %d 0x%lx",size,(void *)remote); > i = ask(req,ans); > if ( i != 0 ) { > free(ans); > @@ -469,7 +469,7 @@ > else > printf("msg%d: Tag desired %d\n",msgid, (int)op->desired_tag); > printf("msg%d: system_buffer %d\n",msgid,op->system_buffer); > - printf("msg%d: Buffer %p\n",msgid,(void *)op->buffer); > + printf("msg%d: Buffer 0x%lx\n",msgid,(void *)op->buffer); > > i = 0; > do { From ashley at pittman.co.uk Thu Nov 5 18:32:43 2009 From: ashley at pittman.co.uk (Ashley Pittman) Date: Thu, 05 Nov 2009 18:32:43 +0000 Subject: [padb] [padb-devel] Simple Makefile patch In-Reply-To: <20091105182033.GL53330@sun.com> References: <20091102172426.GD53330@sun.com> <1257184825.3430.337.camel@alpha> <20091102184534.GE53330@sun.com> <1257188512.3430.343.camel@alpha> <20091102191228.GF53330@sun.com> <1257190832.3430.379.camel@alpha> <20091102195827.GG53330@sun.com> <1257193387.25087.24.camel@alpha> <20091102213052.GH53330@sun.com> <1257199612.25087.54.camel@alpha> <20091105182033.GL53330@sun.com> Message-ID: <1257445963.3633.37.camel@alpha> On Thu, 2009-11-05 at 13:20 -0500, Ethan Mallove wrote: > I don't think any of the printf's fired: You may have to check in the log file to see them. > > I assume it compiles without warnings? > > Right, it compiles w/o warnings. Ashley, -- Ashley Pittman, Bath, UK. Padb - A parallel job inspection tool for cluster computing http://padb.pittman.org.uk From ethan.mallove at sun.com Thu Nov 5 18:48:02 2009 From: ethan.mallove at sun.com (Ethan Mallove) Date: Thu, 5 Nov 2009 13:48:02 -0500 Subject: [padb] [padb-devel] Simple Makefile patch In-Reply-To: <1257445963.3633.37.camel@alpha> References: <20091102184534.GE53330@sun.com> <1257188512.3430.343.camel@alpha> <20091102191228.GF53330@sun.com> <1257190832.3430.379.camel@alpha> <20091102195827.GG53330@sun.com> <1257193387.25087.24.camel@alpha> <20091102213052.GH53330@sun.com> <1257199612.25087.54.camel@alpha> <20091105182033.GL53330@sun.com> <1257445963.3633.37.camel@alpha> Message-ID: <20091105184802.GN53330@sun.com> On Thu, Nov/05/2009 06:32:43PM, Ashley Pittman wrote: > On Thu, 2009-11-05 at 13:20 -0500, Ethan Mallove wrote: > > I don't think any of the printf's fired: > > You may have to check in the log file to see them. None there either: $ cat /tmp/padb-minfo-debug-log-0-VaBwCY req: sym MPIR_dll_name ok 0xff1f9824 zzz: str:35 dmsg Could not find MPIR_dll_name symbol zzz: str:3 exit die -Ethan > > > > I assume it compiles without warnings? > > > > Right, it compiles w/o warnings. > > Ashley, > > -- > > Ashley Pittman, Bath, UK. > > Padb - A parallel job inspection tool for cluster computing > http://padb.pittman.org.uk > From ashley at pittman.co.uk Thu Nov 5 21:28:07 2009 From: ashley at pittman.co.uk (Ashley Pittman) Date: Thu, 05 Nov 2009 21:28:07 +0000 Subject: [padb] [padb-devel] Simple Makefile patch In-Reply-To: <20091105182302.GM53330@sun.com> References: <1257184825.3430.337.camel@alpha> <20091102184534.GE53330@sun.com> <1257188512.3430.343.camel@alpha> <20091102191228.GF53330@sun.com> <1257190832.3430.379.camel@alpha> <20091102195827.GG53330@sun.com> <1257193387.25087.24.camel@alpha> <20091102213052.GH53330@sun.com> <1257199612.25087.54.camel@alpha> <1257440029.3633.20.camel@alpha> <20091105182302.GM53330@sun.com> Message-ID: <1257456487.3633.93.camel@alpha> On Thu, 2009-11-05 at 13:23 -0500, Ethan Mallove wrote: > On Thu, Nov/05/2009 04:53:49PM, Ashley Pittman wrote: > > I had a thought about this last night, it could be that the scanf > > qualifier %p isn't working, googling around for some docs suggest that % > > p won't work, can you try the attached patch to see if that fixes the > > issue? Failing that we might have to wait until I can next get my hands > > on a sun machine. > gcc minfo.c -o minfo.x -ldl -Wall -g > minfo.c: In function 'find_sym': > minfo.c:146: warning: format '%lx' expects type 'long unsigned int *', but argument 3 has type 'void **' > minfo.c:148: warning: format '%lx' expects type 'long unsigned int', but argument 3 has type 'void *' > minfo.c: In function '_find_data': > minfo.c:306: warning: format '%lx' expects type 'long unsigned int', but argument 3 has type 'void *' > minfo.c: In function 'fetch_string': > minfo.c:362: warning: format '%lx' expects type 'long unsigned int', but argument 4 has type 'void *' > minfo.c: In function 'show_op': > minfo.c:468: warning: format '%lx' expects type 'long unsigned int', but argument 3 has type 'void *' Those errors are OK, I get them on Linux as well. If it works I'll add casts to quieten them down. I've booted a Solaris machine here and it seems I was right about this, using %p in sscanf doesn't work on solaris so the patch should work or at least be a step in the right direction. Oddly enough sscanf returns 1 in this case which would seem to indicate it had matched, that code already has a test for the return code of sscanf here but it's not triggering :( Could you run with the patch applied and send me output and the log file generated please. You can just run "padb -aqr0" to show the message queues of rank zero rather than --full-command which will attempt to show you everything about the job. Ashley. -- Ashley Pittman, Bath, UK. Padb - A parallel job inspection tool for cluster computing http://padb.pittman.org.uk From ethan.mallove at sun.com Thu Nov 5 21:36:45 2009 From: ethan.mallove at sun.com (Ethan Mallove) Date: Thu, 5 Nov 2009 16:36:45 -0500 Subject: [padb] [padb-devel] Simple Makefile patch In-Reply-To: <1257456487.3633.93.camel@alpha> References: <1257188512.3430.343.camel@alpha> <20091102191228.GF53330@sun.com> <1257190832.3430.379.camel@alpha> <20091102195827.GG53330@sun.com> <1257193387.25087.24.camel@alpha> <20091102213052.GH53330@sun.com> <1257199612.25087.54.camel@alpha> <1257440029.3633.20.camel@alpha> <20091105182302.GM53330@sun.com> <1257456487.3633.93.camel@alpha> Message-ID: <20091105213645.GQ53330@sun.com> On Thu, Nov/05/2009 09:28:07PM, Ashley Pittman wrote: > On Thu, 2009-11-05 at 13:23 -0500, Ethan Mallove wrote: > > On Thu, Nov/05/2009 04:53:49PM, Ashley Pittman wrote: > > > I had a thought about this last night, it could be that the scanf > > > qualifier %p isn't working, googling around for some docs suggest that % > > > p won't work, can you try the attached patch to see if that fixes the > > > issue? Failing that we might have to wait until I can next get my hands > > > on a sun machine. > > > gcc minfo.c -o minfo.x -ldl -Wall -g > > minfo.c: In function 'find_sym': > > minfo.c:146: warning: format '%lx' expects type 'long unsigned int *', but argument 3 has type 'void **' > > minfo.c:148: warning: format '%lx' expects type 'long unsigned int', but argument 3 has type 'void *' > > minfo.c: In function '_find_data': > > minfo.c:306: warning: format '%lx' expects type 'long unsigned int', but argument 3 has type 'void *' > > minfo.c: In function 'fetch_string': > > minfo.c:362: warning: format '%lx' expects type 'long unsigned int', but argument 4 has type 'void *' > > minfo.c: In function 'show_op': > > minfo.c:468: warning: format '%lx' expects type 'long unsigned int', but argument 3 has type 'void *' > > Those errors are OK, I get them on Linux as well. If it works I'll add > casts to quieten them down. > > I've booted a Solaris machine here and it seems I was right about this, > using %p in sscanf doesn't work on solaris so the patch should work or > at least be a step in the right direction. Oddly enough sscanf returns > 1 in this case which would seem to indicate it had matched, that code > already has a test for the return code of sscanf here but it's not > triggering :( > > Could you run with the patch applied and send me output and the log > file generated please. You can just run "padb -aqr0" to show the > message queues of rank zero rather than --full-command which will > attempt to show you everything about the job. $ padb -aqr0 --config-option rmgr=mpirun --full-report=15961 padb version 3.n (Revision 312) full job report for job 15961 Warning: errors reported by some ranks ======== [0]: Error message from /home/em162155/software/SunOS/sparc/padb/bin/minfo.x: Could not load symbols from dll ======== Warning: errors reported by some ranks ======== [0]: Error message from /home/em162155/software/SunOS/sparc/padb/bin/minfo.x: Could not load symbols from dll ======== Total: 0 communicators, no communication data recorded. ----------------- [0] (1 processes) ----------------- main() at hello_c.c:18 params int argc = '1' [0] char ** argv = '0xffbfe304' [0] locals int rank = '0' [0] int size = '-4201424' [0] ----------------- [0] (1 processes) ----------------- sleep() at ?:? ___nanosleep() at ?:? $ cat /tmp/padb-minfo-debug-log-0-oFPPDR req: sym MPIR_dll_name ok 0xff1f9824 req: string 1024 0xff1f9824 ok /ws/hpc-ct-1/hpc-ct-8.2.1/pkgs/09d/SunOS-10/sparc/built-with-sun/installs/r0Y7/install/lib/openmpi/libompi_dbg_msgq.so zzz: str:31 dmsg Could not load symbols from dll zzz: str:3 exit die $ cat /tmp/padb-minfo-debug-log-0-ZA3VBw req: sym MPIR_dll_name ok 0xff1f9824 req: string 1024 0xff1f9824 ok /ws/hpc-ct-1/hpc-ct-8.2.1/pkgs/09d/SunOS-10/sparc/built-with-sun/installs/r0Y7/install/lib/openmpi/libompi_dbg_msgq.so zzz: str:31 dmsg Could not load symbols from dll zzz: str:3 exit die -Ethan > > Ashley. > > > -- > > Ashley Pittman, Bath, UK. > > Padb - A parallel job inspection tool for cluster computing > http://padb.pittman.org.uk > From ashley at pittman.co.uk Thu Nov 5 21:45:52 2009 From: ashley at pittman.co.uk (Ashley Pittman) Date: Thu, 05 Nov 2009 21:45:52 +0000 Subject: [padb] [padb-devel] Simple Makefile patch In-Reply-To: <20091105213645.GQ53330@sun.com> References: <1257188512.3430.343.camel@alpha> <20091102191228.GF53330@sun.com> <1257190832.3430.379.camel@alpha> <20091102195827.GG53330@sun.com> <1257193387.25087.24.camel@alpha> <20091102213052.GH53330@sun.com> <1257199612.25087.54.camel@alpha> <1257440029.3633.20.camel@alpha> <20091105182302.GM53330@sun.com> <1257456487.3633.93.camel@alpha> <20091105213645.GQ53330@sun.com> Message-ID: <1257457552.3633.100.camel@alpha> On Thu, 2009-11-05 at 16:36 -0500, Ethan Mallove wrote: > $ padb -aqr0 --config-option rmgr=mpirun --full-report=15961 > padb version 3.n (Revision 312) > full job report for job 15961 > > Warning: errors reported by some ranks > ======== > [0]: Error message from /home/em162155/software/SunOS/sparc/padb/bin/minfo.x: Could not load symbols from dll Ok, that's progress. I've managed to boot a VM on amazon and have put sun cluster tools 8.1 (Open-MPI 1.3r19845-ct8.1-b06b-r21) on it, I should be able to get this up and running from this end now. I'll let you know how I get on. Ashley, -- Ashley Pittman, Bath, UK. Padb - A parallel job inspection tool for cluster computing http://padb.pittman.org.uk From padb at googlecode.com Thu Nov 5 22:04:54 2009 From: padb at googlecode.com (padb at googlecode.com) Date: Thu, 05 Nov 2009 22:04:54 +0000 Subject: [padb] r320 committed - Replace %p with %lx in sscanf and 0x%lx in printf calls as ... Message-ID: <000e0cd296c0d31fab0477a6ea1b@google.com> Revision: 320 Author: apittman Date: Thu Nov 5 14:04:01 2009 Log: Replace %p with %lx in sscanf and 0x%lx in printf calls as %p doesn't work on solaris. http://code.google.com/p/padb/source/detail?r=320 Modified: /trunk/src/minfo.c ======================================= --- /trunk/src/minfo.c Tue Nov 3 04:15:34 2009 +++ /trunk/src/minfo.c Thu Nov 5 14:04:01 2009 @@ -141,9 +141,9 @@ if ( i != 0 ) return NULL; - i = sscanf(ans, "%p",&addr); + i = sscanf(ans, "%lx",(long *)&addr); if ( i != 1 ) { - printf("Failed sscanf %d %p\n",i,addr); + printf("Failed sscanf %d 0x%lx\n",i,(long)addr); return NULL; } @@ -301,7 +301,7 @@ if ( size == 0 ) return mqs_ok; - sprintf(req,"data %p %d",(void *)addr,size); + sprintf(req,"data 0x%lx %d",(long)addr,size); i = ask(req,ans); if ( i != 0 ) @@ -360,7 +360,7 @@ char *ans = malloc(size+16); int i; - sprintf(req,"string %d %p",size,(void *)remote); + sprintf(req,"string %d 0x%lx",size,(long)remote); i = ask(req,ans); if ( i != 0 ) { free(ans); @@ -469,7 +469,7 @@ else printf("msg%d: Tag desired %d\n",msgid, (int)op->desired_tag); printf("msg%d: system_buffer %d\n",msgid,op->system_buffer); - printf("msg%d: Buffer %p\n",msgid,(void *)op->buffer); + printf("msg%d: Buffer 0x%lx\n",msgid,(long)op->buffer); i = 0; do { From padb at googlecode.com Thu Nov 5 22:08:57 2009 From: padb at googlecode.com (padb at googlecode.com) Date: Thu, 05 Nov 2009 22:08:57 +0000 Subject: [padb] r321 committed - If dlopen fails to resolve symbols using RTLD_NOW report an error to... Message-ID: <001636d34bfe47d6f50477a6f953@google.com> Revision: 321 Author: apittman Date: Thu Nov 5 14:07:46 2009 Log: If dlopen fails to resolve symbols using RTLD_NOW report an error to the user and retry with RTLD_LAZY. I believe this means the process can die with SIGBUS at any point should the unknown symbol actually be called but that's a problem with the dll and not with minfo. http://code.google.com/p/padb/source/detail?r=321 Modified: /trunk/src/minfo.c ======================================= --- /trunk/src/minfo.c Thu Nov 5 14:04:01 2009 +++ /trunk/src/minfo.c Thu Nov 5 14:07:46 2009 @@ -528,8 +528,13 @@ void *dlhandle; dlhandle = dlopen(filename,RTLD_NOW); - if ( ! dlhandle ) - return -1; + if ( ! dlhandle ) { + show_warning("Unable to dlopen dll with RTLD_NOW, trying LAZY..."); + show_warning(dlerror()); + dlhandle = dlopen(filename,RTLD_LAZY); + if ( ! dlhandle ) + return -1; + } DLSYM(dll_ep,dlhandle,setup_basic_callbacks); DLSYM(dll_ep,dlhandle,setup_image); From padb at googlecode.com Thu Nov 5 22:12:59 2009 From: padb at googlecode.com (padb at googlecode.com) Date: Thu, 05 Nov 2009 22:12:59 +0000 Subject: [padb] r322 committed - Modify gdb_type_offset not to generate warnings when using solaris.... Message-ID: <0016e648fe00b655c20477a707de@google.com> Revision: 322 Author: apittman Date: Thu Nov 5 14:11:00 2009 Log: Modify gdb_type_offset not to generate warnings when using solaris. This is probably a gdb version issue rather than a solaris issue. http://code.google.com/p/padb/source/detail?r=322 Modified: /trunk/src/padb ======================================= --- /trunk/src/padb Wed Nov 4 05:07:25 2009 +++ /trunk/src/padb Thu Nov 5 14:11:00 2009 @@ -5678,10 +5678,13 @@ return gdb_strip_value( $p{reason} ); } +# Add a void * cast here to stop gdb trying to evaluate the type and +# telling us that the offset isn't a valid pointer. Without the cast gdb +# adding extra text after the value which is causing hex to throw warnings. sub gdb_type_offset { my ( $gdb, $type, $field ) = @_; my %p = gdb_send_addr( $gdb, - "-data-evaluate-expression \"&(($type *)0)->$field\"" ); + "-data-evaluate-expression \"(void *)&(($type *)0)->$field\"" ); return unless ( $p{status} eq 'done' ); return hex gdb_strip_value( $p{reason} ); } From padb at googlecode.com Thu Nov 5 22:17:00 2009 From: padb at googlecode.com (padb at googlecode.com) Date: Thu, 05 Nov 2009 22:17:00 +0000 Subject: [padb] r323 committed - Only check for pdsh being installed if we are actually going to use it... Message-ID: <001636e90cfd16f0d70477a7161c@google.com> Revision: 323 Author: apittman Date: Thu Nov 5 14:13:38 2009 Log: Only check for pdsh being installed if we are actually going to use it, for single host jobs where the target processes are local it's not needed. http://code.google.com/p/padb/source/detail?r=323 Modified: /trunk/src/padb ======================================= --- /trunk/src/padb Thu Nov 5 14:11:00 2009 +++ /trunk/src/padb Thu Nov 5 14:13:38 2009 @@ -3039,14 +3039,15 @@ # way can be found in the future. if ( defined $pcmd{host_list} and not defined $pcmd{command} ) { - if ( not find_exe('pdsh') ) { - print - "$conf{rmgr} resource manager requires pdsh to be installed\n"; - return; - } - my @hosts = @{ $pcmd{host_list} }; if ( $hosts[0] ne hostname() or @hosts > 1 ) { + + if ( not find_exe('pdsh') ) { + print +"$conf{rmgr} resource manager on multiple or remote hosts requires pdsh to be installed\n"; + return; + } + $pcmd{require_inner_callback} = 1; my $hlist = join q{,}, @hosts; $pcmd{command} = "pdsh -w $hlist"; From padb at googlecode.com Thu Nov 5 22:29:13 2009 From: padb at googlecode.com (padb at googlecode.com) Date: Thu, 05 Nov 2009 22:29:13 +0000 Subject: [padb] r324 committed - Minor tweaks to get mpirun_get_jobs running on solaris, the --all... Message-ID: <0016e649cd84c956900477a7416a@google.com> Revision: 324 Author: apittman Date: Thu Nov 5 14:28:07 2009 Log: Minor tweaks to get mpirun_get_jobs running on solaris, the --all option now works as expected. http://code.google.com/p/padb/source/detail?r=324 Modified: /trunk/src/padb ======================================= --- /trunk/src/padb Thu Nov 5 14:13:38 2009 +++ /trunk/src/padb Thu Nov 5 14:28:07 2009 @@ -2770,12 +2770,27 @@ map { $mpirun{$_}++ } split $COMMA, $conf{mpirun}; foreach my $pid ( get_process_list($user) ) { + + if ($running_on_solaris) { + my $slink = readlink "/proc/$pid/path/a.out"; + if ( defined $slink ) { + if ( defined $mpirun{ basename($slink) } ) { + push @jobs, $pid; + } + } + next; + } + + # This test does work on solaris but as it reports a full path + # rather than just the basename it'll never pass. It also + # takes a long time to run so skip it. my $name = find_from_status( $pid, 'Name' ); if ( defined $name and defined $mpirun{$name} ) { push @jobs, $pid; next; } + # Linux only. my $link = readlink "/proc/$pid/exe"; next unless defined $link; if ( defined $mpirun{ basename($link) } ) { From padb at googlecode.com Thu Nov 5 22:46:31 2009 From: padb at googlecode.com (padb at googlecode.com) Date: Thu, 05 Nov 2009 22:46:31 +0000 Subject: [padb] r325 committed - Factor out all code dealing with /proc/$pid/exe into one... Message-ID: <0016e64cae6a9c967b0477a77f42@google.com> Revision: 325 Author: apittman Date: Thu Nov 5 14:46:03 2009 Log: Factor out all code dealing with /proc/$pid/exe into one function and modify that to do the right thing on solaris. This fixes several problems to do with finding the executable name. http://code.google.com/p/padb/source/detail?r=325 Modified: /trunk/src/padb ======================================= --- /trunk/src/padb Thu Nov 5 14:28:07 2009 +++ /trunk/src/padb Thu Nov 5 14:46:03 2009 @@ -2771,15 +2771,15 @@ foreach my $pid ( get_process_list($user) ) { - if ($running_on_solaris) { - my $slink = readlink "/proc/$pid/path/a.out"; - if ( defined $slink ) { - if ( defined $mpirun{ basename($slink) } ) { - push @jobs, $pid; - } - } - next; - } + # Works for both solaris and Linux + my $link = proc_link($pid); + if ( defined $link ) { + if ( defined $mpirun{ basename($link) } ) { + push @jobs, $pid; + } + } + + next if ($running_on_solaris); # This test does work on solaris but as it reports a full path # rather than just the basename it'll never pass. It also @@ -2789,13 +2789,6 @@ push @jobs, $pid; next; } - - # Linux only. - my $link = readlink "/proc/$pid/exe"; - next unless defined $link; - if ( defined $mpirun{ basename($link) } ) { - push @jobs, $pid; - } } return @jobs; } @@ -5825,7 +5818,7 @@ } elsif ( $cmd eq 'rank' ) { $res = $vp; } elsif ( $cmd eq 'image' ) { - my $image = readlink "/proc/$gdb->{tracepid}/exe"; + my $image = proc_link( $gdb->{tracepid} ); if ( defined $image ) { $res = $image; } @@ -5894,7 +5887,7 @@ $global{$str_name} = $str_value; if ( $str_name eq 'ihqm' ) { - my $image = readlink "/proc/$gdb->{tracepid}/exe"; + my $image = proc_link( $gdb->{tracepid} ); $str_value =~ s{%s}{$image}; } @@ -8085,13 +8078,24 @@ $handle->{target_response} = undef; return; } + +sub proc_link { + my $pid = shift; + my $path; + if ($running_on_solaris) { + $path = "/proc/$pid/path/a.out"; + } else { + $path = "/proc/$pid/exe"; + } + return readlink $path; +} # Convert from a pid to a command name and do it in a safe manner to avoid # warnings. suid programs tend to have the exe link which is un-readable # so if that yields nothing then load the name from the status file. sub pid_to_name { my $pid = shift; - my $exe = readlink "/proc/$pid/exe"; + my $exe = proc_link($pid); if ( defined $exe ) { return basename($exe); } else { From ashley at pittman.co.uk Thu Nov 5 22:58:12 2009 From: ashley at pittman.co.uk (Ashley Pittman) Date: Thu, 05 Nov 2009 22:58:12 +0000 Subject: [padb] Message queues on solaris Message-ID: <1257461892.3633.120.camel@alpha> I've managed to boot a Solaris 10 machine and take a closer look and have got it working, r325 should work now assuming the dll itself is correct. Below is the output I get from a two process job using sun cluster tools 8.1, what this tells us is two fold: 1) The dll isn't build properly and won't open with dlopen(...,RTLD_NOW), I've added a fallback to use RTLD_LAZY though which allows it to work. 2) The dll is looking for a type called mca_topo_base_comm_1_0_0_t which isn't present in the library and is at that point claiming it cannot display the message queues. These are both bugs in the dll itself and not something I can fix in padb. I've also fixed a number of other problems, mpirun_get_jobs wasn't returning any jobs on solaris so the --all (-a) options wasn't working, this should now behave on Solaris as it does on Linux. root at ip-10-250-13-226:~/padb-read-only/src# ./padb -aq Warning: errors reported by some ranks ======== [0-1]: Error message from ./minfo.x: image_has_queues() failed ======== ---------------- [0-1]: Error string from DLL ---------------- Failed to find some type ---------------- [0-1]: Message from DLL ---------------- mca_topo_base_comm_1_0_0_t ---------------- [0-1]: Warning message from minfo ---------------- Unable to dlopen dll with RTLD_NOW, trying LAZY... ---------------- [0-1]: Warning message from minfo ---------------- ld.so.1: minfo.x: fatal: relocation error: file /opt/SUNWhpc/HPC8.1/sun/lib/openmpi/libompi_dbg_msgq.so: symbol opal_malloc: referenced symbol not found -- Ashley Pittman, Bath, UK. Padb - A parallel job inspection tool for cluster computing http://padb.pittman.org.uk From ethan.mallove at sun.com Fri Nov 6 14:50:35 2009 From: ethan.mallove at sun.com (Ethan Mallove) Date: Fri, 6 Nov 2009 09:50:35 -0500 Subject: [padb] Message queues on solaris In-Reply-To: <1257461892.3633.120.camel@alpha> References: <1257461892.3633.120.camel@alpha> Message-ID: <20091106145034.GR53330@sun.com> On Thu, Nov/05/2009 10:58:12PM, Ashley Pittman wrote: > > I've managed to boot a Solaris 10 machine and take a closer look and > have got it working, r325 should work now assuming the dll itself is > correct. > > Below is the output I get from a two process job using sun cluster tools > 8.1, what this tells us is two fold: > > 1) The dll isn't build properly and won't open with > dlopen(...,RTLD_NOW), I've added a fallback to use RTLD_LAZY though > which allows it to work. > > 2) The dll is looking for a type called mca_topo_base_comm_1_0_0_t which > isn't present in the library and is at that point claiming it cannot > display the message queues. > > These are both bugs in the dll itself and not something I can fix in > padb. > > I've also fixed a number of other problems, mpirun_get_jobs wasn't > returning any jobs on solaris so the --all (-a) options wasn't working, > this should now behave on Solaris as it does on Linux. > > root at ip-10-250-13-226:~/padb-read-only/src# ./padb -aq > Warning: errors reported by some ranks > ======== > [0-1]: Error message from ./minfo.x: image_has_queues() failed > ======== > ---------------- > [0-1]: Error string from DLL > ---------------- > Failed to find some type > ---------------- > [0-1]: Message from DLL > ---------------- > mca_topo_base_comm_1_0_0_t > ---------------- > [0-1]: Warning message from minfo > ---------------- > Unable to dlopen dll with RTLD_NOW, trying LAZY... > ---------------- > [0-1]: Warning message from minfo > ---------------- > ld.so.1: minfo.x: fatal: relocation error: file /opt/SUNWhpc/HPC8.1/sun/lib/openmpi/libompi_dbg_msgq.so: symbol opal_malloc: referenced symbol not found I'm using ClusterTools 8.2.1 (8.1 is about a year old), though I get a similar error: $ padb -aq -Ormgr=mpirun einner: sh: edb: not found Warning: errors reported by some ranks ======== [0]: Error message from /home/em162155/software/SunOS/i386/padb/bin/minfo.x: Could not load symbols from dll ======== ---------------- [0]: Warning message from minfo ---------------- Unable to dlopen dll with RTLD_NOW, trying LAZY... ---------------- [0]: Warning message from minfo ---------------- ld.so.1: minfo.x: fatal: relocation error: file /ws/hpc-ct-1/hpc-ct-8.2.1/pkgs/09d/SunOS-10/i386/built-with-sun/installs/QEjm/install/lib/openmpi/libompi_dbg_msgq.so: symbol opal_mutex_check_locks: referenced symbol not found -Ethan > > > -- > > Ashley Pittman, Bath, UK. > > Padb - A parallel job inspection tool for cluster computing > http://padb.pittman.org.uk > > > _______________________________________________ > padb-devel mailing list > padb-devel at pittman.org.uk > http://pittman.org.uk/mailman/listinfo/padb-devel_pittman.org.uk From ashley at pittman.co.uk Fri Nov 6 15:08:23 2009 From: ashley at pittman.co.uk (Ashley Pittman) Date: Fri, 06 Nov 2009 15:08:23 +0000 Subject: [padb] Message queues on solaris In-Reply-To: <20091106145034.GR53330@sun.com> References: <1257461892.3633.120.camel@alpha> <20091106145034.GR53330@sun.com> Message-ID: <1257520103.3633.148.camel@alpha> On Fri, 2009-11-06 at 09:50 -0500, Ethan Mallove wrote: > I'm using ClusterTools 8.2.1 (8.1 is about a year old) I was just using the version from the package repositories. > though I get a > similar error: > > $ padb -aq -Ormgr=mpirun > einner: sh: edb: not found > Warning: errors reported by some ranks > ======== > [0]: Error message from /home/em162155/software/SunOS/i386/padb/bin/minfo.x: Could not load symbols from dll > ======== > ---------------- > [0]: Warning message from minfo > ---------------- > Unable to dlopen dll with RTLD_NOW, trying LAZY... > ---------------- > [0]: Warning message from minfo > ---------------- > ld.so.1: minfo.x: fatal: relocation error: file /ws/hpc-ct-1/hpc-ct-8.2.1/pkgs/09d/SunOS-10/i386/built-with-sun/installs/QEjm/install/lib/openmpi/libompi_dbg_msgq.so: symbol opal_mutex_check_locks: referenced symbol not found It looks like in this case LAZY opening didn't work either. Again this is likely a problem with the DLL. Can you install this version on the MTT machine (I notice pdsh doesn't have ssh support compiled in their either) and we'll see how SVN code fares? Ashley, -- Ashley Pittman, Bath, UK. Padb - A parallel job inspection tool for cluster computing http://padb.pittman.org.uk From ashley at pittman.co.uk Fri Nov 6 15:58:51 2009 From: ashley at pittman.co.uk (Ashley Pittman) Date: Fri, 06 Nov 2009 15:58:51 +0000 Subject: [padb] [padb-devel] Patch for Support of PBS Pro resource manager In-Reply-To: References: Message-ID: <1257523131.3633.162.camel@alpha> On Fri, 2009-10-30 at 17:06 +0100, thipadin.seng-long at bull.net wrote: > The modified padb has been working fine in our MPI_Bull2 / Pbs Pro > environment. > So I am sending you the patch for you to integrate in the common padb > branch. > I have started from padb 2.5 version, sorry. > If you want to discuss about my coding I am ready, feel free to > comment before you commit it. I've given this a once-over with the changes I think it'll need to work on V3.0 and attached is a patch against r311. I may have mis-understood some of the logic in pbs_find_pids(), I found it hard to follow in places. I hope this is of help to you, it probably won't work as is but might help you with some of the internals. I don't have access to a pbs system to test it on unfortunately. Ashley, -- Ashley Pittman, Bath, UK. Padb - A parallel job inspection tool for cluster computing http://padb.pittman.org.uk -------------- next part -------------- A non-text attachment was scrubbed... Name: padb-pbs.patch Type: text/x-patch Size: 7595 bytes Desc: not available URL: From padb at googlecode.com Mon Nov 9 15:45:25 2009 From: padb at googlecode.com (padb at googlecode.com) Date: Mon, 09 Nov 2009 15:45:25 +0000 Subject: [padb] r326 committed - Minor formatting changes to make the code html compliant. No real new... Message-ID: <0016e64dde580b03b30477f215b2@google.com> Revision: 326 Author: apittman Date: Mon Nov 9 07:45:01 2009 Log: Minor formatting changes to make the code html compliant. No real new content but I can wear a w3.org badge with pride now. http://code.google.com/p/padb/source/detail?r=326 Modified: /trunk/doc/configuration.html /trunk/doc/download.html /trunk/doc/email.html /trunk/doc/extensions.html /trunk/doc/header.html /trunk/doc/index.html /trunk/doc/layout.css /trunk/doc/modes.html /trunk/doc/usage.html ======================================= --- /trunk/doc/configuration.html Thu Sep 10 02:38:12 2009 +++ /trunk/doc/configuration.html Mon Nov 9 07:45:01 2009 @@ -2,39 +2,44 @@

Configuration options

padb has a number of extended configuration options for tuning -it's behavior, these can be set in a number of ways. There are options for -controlling padb itself and how it start up and find jobs and also -mode specific options which control individual (or multiple) modes. +it's behavior, these can be set in a number of ways. There are +options for controlling padb itself and how it start up and +find jobs and also mode specific options which control individual (or +multiple) modes.

-All configuration options have both a name and a value, in some cases the value is a comma separated list. +All configuration options have both a name and a value, in some cases +the value is a comma separated list.

Setting options

Configuration options can be set in the following ways:
    -
  • Command line
    • -Use the command-line option --config-option <key>=<value>. --O can be substituted for --config-option. -
  • Environment
    • -Set the environment variable PADB_<KEY>=<value>. All -environment variables should be specified in upper case. -
  • Config file
    • -Config is also loaded from /etc/padb.conf and ~/.padbrc. Contact the -mailing lists for help with these files. +
  • Command line +
    Use the command-line option --config-option +<key>=<value>. +-O can be substituted for --config-option.
    • +
  • Environment +
    Set the environment +variable PADB_<KEY>=<value>. All environment +variables should be specified in upper case.
    • +
  • Config file +
    Config is also loaded from /etc/padb.conf +and ~/.padbrc. Contact the +mailing lists for help with these files.
-

Viewing active options

-Running padb with the -v option will show both what options are being loaded -and also the used values for all mode specific options. - -

Available options

-A list of configuration options along with their current values can be -obtained by specifying a key of help with or without a value, i.e. padb -Ohelp - -

Example

-In normal use the resource manager is selected automatically based on what is installed, if -for some reason padb cannot determine the resource manager you are using you can -explicitly request one by setting the "rmgr" option. +

Viewing active options

Running padb with the -v +option will show both what options are being loaded and also the used +values for all mode specific options. + +

Available options

A list of configuration options along with +their current values can be obtained by specifying a key of help with +or without a value, i.e. padb -Ohelp + +

Example

In normal use the resource manager is selected +automatically based on what is installed, if for some +reason padb cannot determine the resource manager you are using +you can explicitly request one by setting the "rmgr" option. 
  ashley at alpha:~$ padb --show-jobs
  Error, multiple resource managers detected, use -Ormgr=<resource  
manager>
@@ -49,9 +54,5 @@
  
  
=======================================
--- /trunk/doc/download.html	Tue Oct  6 11:49:59 2009
+++ /trunk/doc/download.html	Mon Nov  9 07:45:01 2009
@@ -37,9 +37,5 @@
  
  
=======================================
--- /trunk/doc/email.html	Mon Sep 14 01:56:33 2009
+++ /trunk/doc/email.html	Mon Nov  9 07:45:01 2009
@@ -14,9 +14,5 @@
  
  
=======================================
--- /trunk/doc/extensions.html	Thu Sep 10 02:38:12 2009
+++ /trunk/doc/extensions.html	Mon Nov  9 07:45:01 2009
@@ -116,9 +116,5 @@
  
  
=======================================
--- /trunk/doc/header.html	Tue Oct  6 11:49:59 2009
+++ /trunk/doc/header.html	Mon Nov  9 07:45:01 2009
@@ -1,3 +1,4 @@
+
  
    
    Padb: A parallel debugging tool
@@ -12,21 +13,18 @@
  
  
=======================================
--- /trunk/doc/index.html	Tue Oct  6 09:38:43 2009
+++ /trunk/doc/index.html	Mon Nov  9 07:45:01 2009
@@ -1,12 +1,12 @@
  

  
About

-
Padb is a Job Inspection Tool for examining and
-debugging parallel programs, primarily it simplifies the process of
-gathering stack traces on compute clusters however it also supports a wide  
range of other functions.
-Padb supports a number of parallel environments and it works  
out-of-the-box on the majority of
-clusters.  It's an
-open source, non-interactive, command line, script-able tool intended
-for use by programmers and system administrators alike.

+
Padb is a Job Inspection Tool for examining and debugging
+parallel programs, primarily it simplifies the process of gathering stack  
traces
+on compute clusters however it also supports a wide range of other  
functions.
+Padb supports a number of parallel environments and it works
+out-of-the-box on the majority of clusters. It's an open source,
+non-interactive, command line, script-able tool intended for use by  
programmers
+and system administrators alike.


  

  Padb is developed and maintained by
@@ -15,22 +15,22 @@
  

  


-
Recent News

+
Recent News


  
    
- 
  • 06-10-09: Final 3.0 release avaliable for  
download
- 
  • 15-09-09: 3.0-rc2 avaliable for download
- 
  • 01-09-09: A 3.0-rc release is avaliable to download from the
-  downloads page.
- 
  • 25-06-09: A 2.5 stable release (version 2.5) is avaliable to  
download from the
-  downloads page.
-  
    • 
+ 
  • 06-10-09: 3.0 release avaliable for
+ download.
+ 
  • 15-09-09: 3.0-rc2 avaliable for download.
+ 
  • 01-09-09: A 3.0-rc release is avaliable to download from
+ the downloads page.
+ 
  • 25-06-09: A 2.5 stable release (version 2.5) is avaliable to
+ download from the downloads page.
  

  

  


-
Features

-The following modes of operation are supported:
+
Features
 The following modes of  
operation are
+supported:
  
    
  
  • Stack trace generation
    • 
  
  • MPI Message queue display
    • 
@@ -42,13 +42,12 @@
  


  
What padb can't do

-Padb is a job inspection tool, it can tell you want you want
-to know about your job and your MPI stack, it will not, however, tell
-you about your cluster as a whole and it won't diagnose problems
-with your wider environment, including you job launcher or runtime
-environment.  Padb
-does not launch or wrap your jobs for you, it is not a job harness but
-rather attaches to or targets jobs which are already running.
+Padb is a job inspection tool, it can tell you want you want to know
+about your job and your MPI stack, it will not, however, tell you about  
your
+cluster as a whole and it won't diagnose problems with your wider  
environment,
+including you job launcher or runtime environment. Padb does not  
launch
+or wrap your jobs for you, it is not a job harness but rather attaches to  
or
+targets jobs which are already running.

  
License

  padb is licensed under the
@@ -57,42 +56,36 @@

  
History

  Padb was originally conceived by software developers at
-Quadrics around 2004 to solve the
-kind of problems facing them at the time.  It's been a part of the
-Quadrics software stack for a number of years and has recently been
-made available to a wider audience.  It has been commercially supported
-for a number of years and is known to work at a scale of tens of thousands  
of
-processes.
+Quadrics around 2004 to solve the  
kind of
+problems facing them at the time. It's been a part of the Quadrics software
+stack for a number of years and has recently been made available to a wider
+audience. It has been commercially supported for a number of years and is  
known
+to work at a scale of tens of thousands of processes.

  
Parallel Environments

-Padb works and is supported on the following parallel
-environments and MPI stacks.  Not all features are available on all
-runtimes.
+Padb works and is supported on the following parallel environments  
and
+MPI stacks. Not all features are available on all runtimes.
  
    
-
  • Quadrics RMS
    • 
-Runs natively on clusters running RMS.
-
  • Slurm
    • 
-Runs natively on clusters running Slurm irrespective of the runtime used.
-
  • Open MPI
    • 
-Supports orte or OpenMPI jobs run under slurm.
-
  • MPICH2
    • 
-Supports mpd or MPICH2 jobs run under slurm.
+
  • Quadrics RMS
    Runs natively on
+clusters running RMS.
    • 
+
  • Slurm
    Runs
+natively on clusters running Slurm irrespective of the runtime used.
    • 
+
  • Open MPI
    Supports orte or
+OpenMPI jobs run under slurm.
    • 
+
  • MPICH2
    Supports mpd  
or
+MPICH2 jobs run under slurm.
    • 
  

  
In addition padb can be told to target individual UNIX processes.

  
Prerequisites

-Padb requires very little support from the OS or parallel
-environment to run, it's main use is to assist in the debugging of
-parallel applications, it's therefore assumed that you have a working MPI
-stack or other parallel environment and that "Hello world" application  
runs to completion without
-error.
  A Linux operating system is assumed and a working
+Padb requires very little support from the OS or parallel  
environment to
+run, it's main use is to assist in the debugging of parallel applications,  
it's
+therefore assumed that you have a working MPI stack or other parallel
+environment and that "Hello world" application runs to completion without
+error.
 A Linux operating system is assumed and a working
  gdb is required for stack trace functionality.

  

  
=======================================
--- /trunk/doc/layout.css	Thu Sep 10 02:38:12 2009
+++ /trunk/doc/layout.css	Mon Nov  9 07:45:01 2009
@@ -32,6 +32,8 @@

  #footer {
   clear: both;
+ font-size: 75%;
+ text-align: center;
  }

  pre.code {
@@ -42,3 +44,10 @@
  border: 1px solid;
  }

+span.menu_heading {
+font-size: 110%;
+}
+
+span.menu_subheading {
+font-size: 80%;
+}
=======================================
--- /trunk/doc/modes.html	Mon Sep 14 01:56:33 2009
+++ /trunk/doc/modes.html	Mon Nov  9 07:45:01 2009
@@ -49,7 +49,7 @@
  
  $ padb --all --proc-summary
  

-
+
  
  
  vpid  hostname  pid   vmsize     vmrss    S  %cpu  command
@@ -62,7 +62,7 @@
     1        i1  2614  160336 kB  4468 kB  R    47    a.out
     0     fnarp  2788  160336 kB  4464 kB  R    44    a.out
  

-
+
  

  The config option proc-sort-key controls which column the table is  
sorted by, the default is vpid.
@@ -80,7 +80,7 @@
  
  $ padb --all --rank=0 --proc-info
  

-
+
  
  
  hostname:fnarp
@@ -138,7 +138,7 @@
  fd29: pipe:[56966] (0 01)
  pcpu: 47
  

-
+
  

  

@@ -153,7 +153,7 @@
  
  $ padb --all --stack-trace --tree
  

-
+
  
  
  -----------------
@@ -186,7 +186,7 @@
            ompi_request_default_wait_all() at ?:?
              opal_progress() at ?:?
  

-
+
  

  The config options stack-shows-locals and stack-shows-params  
can be enabled to display more information in the stack
@@ -197,7 +197,7 @@
  
  $ padb --any --stack-trace --rank 0 -O stack-shows-locals=1  
-Ostack-shows-params=1
  

-
+
  
  
  main() at ?:?
@@ -260,7 +260,7 @@
  sched_yield() at ?:?
  __kernel_vsyscall() at ?:?
  

-
+
  

  

@@ -287,7 +287,7 @@
  
  $ padb --all --compress --mpi-queue
  

-
+
  
  
  ----------------
@@ -357,7 +357,7 @@
  comm5: size: '1'
  comm5: id: '0x1c'
  

-
+
  

  

@@ -371,7 +371,7 @@
  
  $ padb --all --deadlock
  

-
+
  
  
  Information for group '0x4' (MPI COMMUNICATOR 4 DUP FROM 0)
@@ -384,7 +384,7 @@
  Group members [1-3] (size 4) are not in a call to the collectives.
  Total: 6 groups of which 2 are in use.
  

-
+
  

  

@@ -418,7 +418,6 @@
  
  $ padb --all --mpi-watch --watch -Owatch-clears-screen=no
  

-
  
  
  u: unexpected messages U: unexpected and other messages
@@ -718,14 +717,10 @@
  ,,,RRRRRRRR,,,,,bbbb
  b*bb*****b*b*bbbbbbb
  

-
+
  
  

  
=======================================
--- /trunk/doc/usage.html	Mon Sep 14 01:56:33 2009
+++ /trunk/doc/usage.html	Mon Nov  9 07:45:01 2009
@@ -5,13 +5,13 @@
   
   
Selecting the Resource Manager (Job Launcher)

  

-Padb supports many resource managers and should select
-the appropriate one for your cluster, if you have more than
-one resource manager installed or padb can't detect
-the correct one use the rmgr configuration option.
-
-
If no resource manager is found you can use -O rmgr=local and  
process
-identifiers (pids) will be used instead of job ids.
+Padb supports many resource managers and should select the
+appropriate one for your cluster, if you have more than one resource
+manager installed or padb can't detect the correct one use
+the rmgr configuration option.
+
+
If no resource manager is found you can use -O rmgr=local
+and process identifiers (pids) will be used instead of job ids.


  

@@ -22,6 +22,15 @@
    Notes
   
   
+  Any
+  mpirun
+  Works with any resource manager or software stack that is
+  compliant with
+  the MPI
+  debugger interface.  It is recommended to use support for your
+  specific resource manager if it exists.
+ 
+ 
    Slurm
    slurm
    Fully supported
@@ -44,87 +53,93 @@
   
    None
    local
-  allows process identifiers (pids) to be specified as jobids and  
treats that process
-as a single process job.
+  allows process identifiers (pids) to be specified as jobids and
+treats that process as a single process job.
   
   
    None
    local-fd
-  as local but only selects processes which have a certain file open,  
use the "local-fd-name" configuration option to set the filename.
+  as local but only selects processes which have a certain file
+  open, use the "local-fd-name" configuration option to set the
+  filename.
   
   
    None
    local-qsnet
-  as local-fd with local-fd-name set to /proc/qsnet/elan/user to  
automatially select network jobs on the local node.
+  as local-fd with local-fd-name set to /proc/qsnet/elan/user to
+  automatially select network jobs on the local node.
   
  
  


-
The --list-rmgrs option can be used to show a list of detected
-resource managers and their active jobs.
+
The --list-rmgrs option can be used to show a list of
+detected resource managers and their active jobs.

  

   
   
Selecting the job(s) to target

  

-Padb provides several ways for the user to select which jobs
-to target depending on requirements.  All jobs detected by padb are
+Padb provides several ways for the user to select which jobs to
+target depending on requirements.  All jobs detected by padb are
  identified by a unique numeric "jobid", typically this maps directly
-only a identifier provided by the underlying resource manager.
-The default is to target jobs of the current user, this can be
-over-ridden with the --user flag.
+only a identifier provided by the underlying resource manager.  The
+default is to target jobs of the current user, this can be over-ridden
+with the --user flag.

  
Showing list of current jobs

  To show a list of currently running jobs for a given user use the
---show-jobs option.  Alternatively the --list-rmgrs option
-shows a list of detected resource managers along with a list of currently  
active jobs for each of them.
-
-
To target all jobs

-To target all jobs currently running for a given user use the --all  
(-a) flag.
-
-
To target any jobs

-To target "any" job currently running for a given user use the  
--any (-A) flag.
-This differs from targeting all jobs as it will exit with an error if more
-than one job is running.
-
-
To target a specific job

-To target a specific job specify the jobid for the job on the command line,
-after all other options.
+--show-jobs option.  Alternatively the --list-rmgrs
+option shows a list of detected resource managers along with a list of
+currently active jobs for each of them.
+
+
To target all jobs
 To target all jobs currently running for a
+given user use the --all (-a) flag.
+
+
To target any jobs
 To target "any" job currently running for
+a given user use the --any (-A) flag.  This differs from
+targeting all jobs as it will exit with an error if more than one job
+is running.
+
+
To target a specific job
 To target a specific job specify the
+jobid for the job on the command line, after all other options.

  

- 
+ 
   
To target a specific job (Full report option)

  

  If trying to diagnose a problem or gather information there is another
-option, --full-report=<jobid>, this tells padb to target the  
job
-specified and to report all information about the job it knows how to
-collect.  This option is typically used when creating bug reports to
-send to third parties or to inspect a job for anomalies.
+option, --full-report=<jobid>, this tells padb to target
+the job specified and to report all information about the job it knows
+how to collect.  This option is typically used when creating bug
+reports to send to third parties or to inspect a job for anomalies.

  


-
Selecting ranks (Processes)

-In modes where data for each process is reported separately it is possible
-to restrict which ranks are queried, this is done via the --rank  
option.
-Multiple ranks can be selected by specifying --rank multiple times.
+
Selecting ranks (Processes)
 In modes where
+data for each process is reported separately it is possible to
+restrict which ranks are queried, this is done via the --rank
+option.  Multiple ranks can be selected by specifying --rank
+multiple times.

  


  
Selecting which mode to run in.

  Padb can present an array of different information about your
  select jobs and it can present it in a number of different ways.  With
-the exception of Full Report only one mode can be selected, if you need
-more information about the program run padb more than once.
+the exception of Full Report only one mode can be selected, if you
+need more information about the program run padb more than
+once.

  

   
   
Full Report

  

  If you are just starting with padb or are creating an error
-report for somebody else then the --full-report=<jobid>  
option is a good place
-to start, this will complete a full diagnostic report for the job
-iterating over the more common padb options.  For large jobs this can
-generate a lot of output so redirecting to a file is recommended.
+report for somebody else then the --full-report=<jobid>
+option is a good place to start, this will complete a full diagnostic
+report for the job iterating over the more common padb options.  For
+large jobs this can generate a lot of output so redirecting to a file
+is recommended.

  


@@ -133,9 +148,5 @@
  

  



From padb at googlecode.com  Thu Nov 12 11:55:56 2009
From: padb at googlecode.com (padb at googlecode.com)
Date: Thu, 12 Nov 2009 11:55:56 +0000
Subject: [padb] r327 committed - Don't require the use of an inner callback
	when simply targetting loca...
Message-ID: <000e0cd20f80dcd73e04782b3934@google.com>

Revision: 327
Author: apittman
Date: Thu Nov 12 03:54:49 2009
Log: Don't require the use of an inner callback when simply targetting local
processes, we can just use stdout of the child process to pass back
the port number.  This only works since I added the flush to the child
process.

http://code.google.com/p/padb/source/detail?r=327

Modified:
  /trunk/src/padb

=======================================
--- /trunk/src/padb	Thu Nov  5 14:46:03 2009
+++ /trunk/src/padb	Thu Nov 12 03:54:49 2009
@@ -421,28 +421,25 @@
  };

  $rmgr{local} = {
-    get_active_jobs        => \&local_get_jobs,
-    job_is_running         => \&local_job_is_running,
-    setup_pcmd             => \&local_setup_pcmd,
-    find_pids              => \&local_find_pids,
-    require_inner_callback => 1,
+    get_active_jobs => \&local_get_jobs,
+    job_is_running  => \&local_job_is_running,
+    setup_pcmd      => \&local_setup_pcmd,
+    find_pids       => \&local_find_pids,
  };

  $rmgr{'local-qsnet'} = {
-    is_installed           => \&local_q_is_installed,
-    get_active_jobs        => \&local_q_get_jobs,
-    job_is_running         => \&local_job_is_running,
-    setup_pcmd             => \&local_setup_pcmd,
-    inner_rmgr             => 'local',
-    require_inner_callback => 1,
+    is_installed    => \&local_q_is_installed,
+    get_active_jobs => \&local_q_get_jobs,
+    job_is_running  => \&local_job_is_running,
+    setup_pcmd      => \&local_setup_pcmd,
+    inner_rmgr      => 'local',
  };

  $rmgr{'local-fd'} = {
-    get_active_jobs        => \&local_fd_get_jobs,
-    job_is_running         => \&local_job_is_running,
-    setup_pcmd             => \&local_setup_pcmd,
-    inner_rmgr             => 'local',
-    require_inner_callback => 1,
+    get_active_jobs => \&local_fd_get_jobs,
+    job_is_running  => \&local_job_is_running,
+    setup_pcmd      => \&local_setup_pcmd,
+    inner_rmgr      => 'local',
  };

   
###############################################################################



From padb at googlecode.com  Thu Nov 12 14:51:23 2009
From: padb at googlecode.com (padb at googlecode.com)
Date: Thu, 12 Nov 2009 14:51:23 +0000
Subject: [padb] r328 committed - If showing the message queues via the -q
	option the spec says to try...
Message-ID: <0016e64eeca658a5b604782dadf9@google.com>

Revision: 328
Author: apittman
Date: Thu Nov 12 06:51:01 2009
Log: If showing the message queues via the -q option the spec says to try
edb (a QsNet program) and fall back to minfo if that doesn't work.
Add a test for edb being installed and if it's not then go straight to
minfo.  This should prevent errors seen on Solaris where the shell
complains edb isn't found.

http://code.google.com/p/padb/source/detail?r=328

Modified:
  /trunk/src/padb

=======================================
--- /trunk/src/padb	Thu Nov 12 03:54:49 2009
+++ /trunk/src/padb	Thu Nov 12 06:51:01 2009
@@ -7550,6 +7550,14 @@

  sub show_queue {
      my ( $carg, $vp, $pid ) = @_;
+
+    # If edb isn't installed (this isn't a Quadrics system) don't even try
+    # and run edb as it'll fail, switch straight to minfo for the MPI
+    # message queues instead.
+    if ( !find_exe( $inner_conf{edb} ) ) {
+        show_mpi_queue( $carg, $vp, $pid );
+        return;
+    }

      # Nobble the LD_LIBRARY_PATH to give etrace the best chance of working.
      my %remote_env = get_remote_env($pid);



From padb at googlecode.com  Thu Nov 12 15:15:42 2009
From: padb at googlecode.com (padb at googlecode.com)
Date: Thu, 12 Nov 2009 15:15:42 +0000
Subject: [padb] r329 committed - Add a long comment about resource managers
	and the possibilty...
Message-ID: <0016e64af8a841abcc04782e0437@google.com>

Revision: 329
Author: apittman
Date: Thu Nov 12 07:14:50 2009
Log: Add a long comment about resource managers and the possibilty
of using a different resource manager for launching the shadow
job than the one the target job is using.

http://code.google.com/p/padb/source/detail?r=329

Modified:
  /trunk/src/padb

=======================================
--- /trunk/src/padb	Thu Nov 12 06:51:01 2009
+++ /trunk/src/padb	Thu Nov 12 07:14:50 2009
@@ -371,6 +371,35 @@
  # require_inner_callback var n/a no       Resource manager doesn't  
preserve line
  #                                         ordering of stdout.

+# Current a single resource manager is assumed which is used for (a)
+# discovering jobs (b) launching the shadow job and (c) finding the target
+# processes from the inner padb processes.  Two caveats to this exist, the
+# "inner_rmgr" setting which allows a resource manager which has specifed
+# (a) and (b) to pass the buck onto a different resource manager for (c).
+# This is typically used for schedulers or software layers which sit on top
+# of the resource manager.  Care need to be taken in this case to convert
+# the jobid when switching from outer to inner (only lsf-rms does this
+# currently and I'm not 100% sure that still works).  Also the setup_job()
+# callback allows resource managers which provide (a) to not provide (b)
+# but to rely on padb to launch a shadow job on the host-list it provides.
+# Padb uses pdsh for this.
+#
+# What would be possible however is to split (b) off completely, many
+# resource managers launch the shadow job simply by taking a hostlist so it
+# would be possible to mix-and-match (a) and (b) from different resource
+# managers, perhaps use mpd to query the job, return a host list and then
+# use orte to launch the actual job.
+#
+# For resource managers which don't provide (b) (currently mpirun only but
+# expected to grow) padb uses pdsh which is limited in the size of job that
+# it can run, one solution to this might be to require say a open-mpi
+# install and use orterun to launch the shadow job.  This could have
+# benefits elsewhere as well, both the speed of (b) and it's ability to
+# interact with padb (for port number forwarding) are crucial for
+# scalability, having a single stack for padb to sit on would allow
+# concentration of tuning effort in a single place which is something  
everyone
+# could benefit from.
+
  my %rmgr;

  $rmgr{mpirun} = {



From thipadin.seng-long at bull.net  Tue Nov 17 16:17:03 2009
From: thipadin.seng-long at bull.net (thipadin.seng-long at bull.net)
Date: Tue, 17 Nov 2009 17:17:03 +0100
Subject: [padb] =?iso-8859-1?q?R=E9f=2E_=3A_Re=3A_=5Bpadb-devel=5D_Patch_f?=
 =?iso-8859-1?q?or_Support_of_PBS__Pro_resource_manager?=
Message-ID: 

Hi,
I have merged your patch against r311, and been testing the overall 
version.
I am doing a kind of step by step, then I not am going through all the 
code,
bugs met are:

1-Loop in a while instruction:

2620 sub pbs_get_lqsub {
2621     my ( $user, $server ) = @_;
2622     my $job;
2623     my $cmd = "qstat -w -n -u $user \@$server";
2624 
2625     my @output = slurp_cmd($cmd);
2626     while (@output) {
2627         if (/\d+\.$server/i) {
.
.
.
.
it issues some kinds of messages plenty on the screen:

Use of uninitialized value in pattern match (m//) at ./padb line 2627.
 at ./padb line 2627
        main::pbs_get_lqsub('thipa', 'xn0') called at ./padb line 2680
        main::pbs_get_data('thipa') called at ./padb line 2688
        main::pbs_get_jobs('thipa') called at ./padb line 3109
        main::job_is_running('27611.xn0', 'thipa') called at ./padb line 
5183
        main::outer_main() called at ./padb line 9048
Use of uninitialized value in pattern match (m//) at ./padb line 2627.
 at ./padb line 2627
        main::pbs_get_lqsub('thipa', 'xn0') called at ./padb line 2680
        main::pbs_get_data('thipa') called at ./padb line 2688
        main::pbs_get_jobs('thipa') called at ./padb line 3109
        main::job_is_running('27611.xn0', 'thipa') called at ./padb line 
5183
        main::outer_main() called at ./padb line 9048
Use of uninitialized value in pattern match (m//) at ./padb line 2627.
 at ./padb line 2627
        main::pbs_get_lqsub('thipa', 'xn0') called at ./padb line 2680
        main::pbs_get_data('thipa') called at ./padb line 2688
        main::pbs_get_jobs('thipa') called at ./padb line 3109
        main::job_is_running('27611.xn0', 'thipa') called at ./padb line 
5183
        main::outer_main() called at ./padb line 9048
:pbs_get_data('thipa') called at ./padb line 2688
        main::pbs_get_jobs('thipa') called at ./padb line 3109
        main::job_is_running('27611.xn0', 'thipa') called at ./padb line 
5183
        main::outer_main() called at ./padb line 9048

 I have to break on the screen to get prompt:
So I guess it is a infinite loop.
I have changed  'while(@output)'  for 'foreach(@output)', to correct this probleme.

2- Job is not found:

So when the loop is disappeared  i can go further:

./padb -O rmgr=pbs -tx 27611.xn0
Job 27611.xn0 is not active
[thipa at xn5]$ qstat
Job id            Name             User              Time Use S Queue
----------------  ---------------- ----------------  -------- - -----
27611.xn0         STDIN            thipa             00:00:06 R workq  
[thipa at xn5]$

The jobs that are display by qstat have the suffice with .xn0 (which is 
the server), 
so we used to pick up the whole job id as input jobid.
So something have to be changed (code or synopsis).

I am waiting for your patch (or reply) to continue.
Thipadin.
More later.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: 

From ashley at pittman.co.uk  Wed Nov 18 08:00:54 2009
From: ashley at pittman.co.uk (Ashley Pittman)
Date: Wed, 18 Nov 2009 08:00:54 +0000
Subject: [padb]
 =?iso-8859-1?q?R=E9f=2E_=3A_Re=3A_=5Bpadb-devel=5D_Patch_f?=
 =?iso-8859-1?q?or_Support_of_PBS_Pro_resource_manager?=
In-Reply-To: 
References: 
Message-ID: <1258531254.3377.20.camel@alpha>

On Tue, 2009-11-17 at 17:17 +0100, thipadin.seng-long at bull.net wrote:
>  I have to break on the screen to get prompt: 
> So I guess it is a infinite loop. 
> I have changed  'while(@output)'  for 'foreach(@output)', to correct
> this probleme.

That looks like a simple mistake on my part.  I prefer not to use $_ in
my code (either implicitly or explicitly) as I think it makes it less
readable, once the code works it's easy enough to make variables
explicit however, I just didn't do it in the patch because I prefer to
change as least as I can possibly get away with unless I can test it
immediately.

> 2- Job is not found: 
> 
> So when the loop is disappeared  i can go further: 
> 
> ./padb -O rmgr=pbs -tx 27611.xn0 
> Job 27611.xn0 is not active 
> [thipa at xn5]$ qstat 
> Job id            Name             User              Time Use S Queue 
> ----------------  ---------------- ----------------  -------- - ----- 
> 27611.xn0         STDIN            thipa             00:00:06 R workq

>            
> [thipa at xn5]$ 
> 
> The jobs that are display by qstat have the suffice with .xn0 (which
> is the server), 
> so we used to pick up the whole job id as input jobid. 
> So something have to be changed (code or synopsis).

I guess the job id as you requested it (27611.xn0) does not match what
it returned by pbs_get_jobs(), there is a problem here to do with the
server.  In the past all job id's have been numeric, this hasn't been a
problem but isn't something that I've strived for, it's just that so far
all resource managers have worked that way so that's how I think of it.
There is no technical reason for this to be true however so how about we
just say that in the future jobid's have to be alphanumeric strings,
this would work in this case although would have the downside you
couldn't specify the job as 27611 in the case above.

padb --show-jobs should show you what padb thinks the job id's are and
of course using -a rather than specifying a job tells it to use all jobs
so it'll just attempt to target one in the case above, regardless of
what it thinks it's called.

I'd be happy for a patch supporting either implementation, i.e. I don't
have a strong preference either way.   You can either have the jobid
encompass both the number and the server or you could continue with what
I attempted to encode in the patch I sent you where the job id is the
number and the server becomes a configuration option.

Actually this could make life easier for slurm and the way it handles
job steps, it effectively appends ".0" to the padb job id before handing
it over to slurm so this could probably be simplified if the .0 became a
optional part of the job id itself rather than a separate configuration
option.

> I am waiting for your patch (or reply) to continue.

I hope this helps you along the way, I can't really code anything from
here as I don't have access to a pbs system.

Ashley,

-- 

Ashley Pittman, Bath, UK.

Padb - A parallel job inspection tool for cluster computing
http://padb.pittman.org.uk




From thipadin.seng-long at bull.net  Wed Nov 18 15:48:50 2009
From: thipadin.seng-long at bull.net (thipadin.seng-long at bull.net)
Date: Wed, 18 Nov 2009 16:48:50 +0100
Subject: [padb] =?iso-8859-1?q?R=E9f=2E_=3A_Re=3A_R=E9f=2E_=3A_Re=3A_=5Bpa?=
 =?iso-8859-1?q?db-devel=5D_Patch__for_Support_of_PBS_Pro_resource_manager?=
Message-ID: 

Hi,

I've got through padb test now (assuming jobid is just numeric part), the 
result is OK, and I met 2 bugs:

1- Path of remote padb:

[thipa at xn5 padb_open]$ ./padb -O rmgr=pbs -tx 27611
einner: xn20: bash: ./padb: No such file or directory
einner: pdsh at xn5: xn20: ssh exited with exit code 127
einner: xn19: bash: ./padb: No such file or directory
einner: pdsh at xn5: xn19: ssh exited with exit code 127
Unexpected EOF from Inner stdout (connecting)
Unexpected EOF from Inner stderr (connecting)
Waiting for signon from 2 hosts.
Unexpected exit from parallel command (state=connecting)
[thipa at xn5 padb_open]$

[thipa at xn20 padb_open]$ ssh -V
OpenSSH_5.1p1, OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008

I think from OpenSSH_5.1, the remote SSH does not execute .bashrc anymore
So PATH variable is not set from your .bashrc,
As the consequence path is not found,
So path to remote host must be a full path, I did the patch as follows:

sub pbs_setup_job {

.
.

    my $pwd=$ENV{PWD};
    my $dirnm = dirname ($0);
    my $base = basename ($0);
    # if padb is launch as padb then dirname is .
    # if padb is launched with a full path then dir is full
    my $out;
    if ($dirnm eq ".") {
       $out=" $pwd\/$base ";
    } else {
       $out=" $0 ";
    }
    $pcmd{padb_path} = $out;

.
}

sub  go_job {

.

my $padb_path  = $pcmd{padb_path};

.
.
 #replace this line $cmd .= " $0 --inner"; by

    if (!defined $padb_path) {
       $cmd .= " $0 --inner";
    } else {
       $cmd .= " $padb_path --inner ";
    }

.

.

}

If you have another idea i take it.


2- Use of uninitialized value in subtraction (-) at ./padb line 4077

[thipa at xn5 padb_open]$ ./padb -O rmgr=pbs -O stack-shows-locals=no  -O 
stack-shows-params=no -tx 27611
Use of uninitialized value in subtraction (-) at ./padb line 4077.
-----------------
[0-1,3,6] (4 processes)
-----------------
main() at pp_sndrcv_spbl.c:50
  PMPI_Finalize() at ?:?
    MPID_Finalize() at ?:?
      MPIDI_CH3_Progress_wait() at ?:?
        MPIDU_Sock_wait() at ?:?
          poll() at ?:?
-----------------
[2,4] (2 processes)
-----------------
ThreadId: 1
  -----------------
  [2] (1 processes)
  -----------------
  main() at pp_sndrcv_spbl.c:46
    PMPI_Recv() at ?:?
      MPID_Progress_wait() at ?:?
        MPIDI_CH3_Progress_wait() at ?:?
          MPIDU_Sock_wait() at ?:?
            poll() at ?:?
              ThreadId: 2
                start_thread() at ?:?
                  fd_server() at server.c:354
                    select() at ?:?
  -----------------
  [4] (1 processes)
  -----------------
  main() at pp_sndrcv_spbl.c:50
    PMPI_Finalize() at ?:?
      MPID_Finalize() at ?:?
        MPIDI_CH3_Progress_wait() at ?:?
          MPIDU_Sock_wait() at ?:?
            poll() at ?:?
              ThreadId: 2
                start_thread() at ?:?
                  fd_server() at server.c:354
                    select() at ?:?
[thipa at xn5 padb_open]$ 

line 4077 looks like:

4058 sub check_signon {
4059     my ( $comm_data, $data ) = @_;
4060     return if ( $conf{check_signon} eq 'none' );
.
.
4075     my $rng = rng_create_empty();
4076 
4077     foreach my $proc ( 0 .. $comm_data->{nprocesses} - 1 ) {
4078         if ( not defined $here{$proc} ) {
4079             rng_add_value( $rng, $proc );
4080         }
4081     }

3- Question about starting inner padb:


How can I start an inner padb by hand on a remote host to debug such as:
perl -d ./padb --inner --jobid=27611.xn0 --stack-trace -O rmgr="pbs" 
--line-formatted
like I did it before, because this command doesn't work anymore.
You have changed it with "call back" and communication on ports.


Here is the diff again r311 (diff r311 newone).



So you can integrate my new patch and try to correct the point 2,
and send me back the new one, i will test it over.

Regards,
Thipadin.








Ashley Pittman 
11/18/2009 09:00 AM

 
        Pour :  thipadin.seng-long at bull.net
        cc :    florence.vallee at bull.net, francois.wellenreiter at bull.net, 
padb-devel at pittman.org.uk, Sylvain.JEAUGEY at bull.net
        Objet : Re: R?f. : Re: [padb-devel] Patch for Support of PBS Pro resource manager

On Tue, 2009-11-17 at 17:17 +0100, thipadin.seng-long at bull.net wrote:
>  I have to break on the screen to get prompt: 
> So I guess it is a infinite loop. 
> I have changed  'while(@output)'  for 'foreach(@output)', to correct
> this probleme.

That looks like a simple mistake on my part.  I prefer not to use $_ in
my code (either implicitly or explicitly) as I think it makes it less
readable, once the code works it's easy enough to make variables
explicit however, I just didn't do it in the patch because I prefer to
change as least as I can possibly get away with unless I can test it
immediately.

> 2- Job is not found: 
> 
> So when the loop is disappeared  i can go further: 
> 
> ./padb -O rmgr=pbs -tx 27611.xn0 
> Job 27611.xn0 is not active 
> [thipa at xn5]$ qstat 
> Job id            Name             User              Time Use S Queue 
> ----------------  ---------------- ----------------  -------- - ----- 
> 27611.xn0         STDIN            thipa             00:00:06 R workq

> 
> [thipa at xn5]$ 
> 
> The jobs that are display by qstat have the suffice with .xn0 (which
> is the server), 
> so we used to pick up the whole job id as input jobid. 
> So something have to be changed (code or synopsis).

I guess the job id as you requested it (27611.xn0) does not match what
it returned by pbs_get_jobs(), there is a problem here to do with the
server.  In the past all job id's have been numeric, this hasn't been a
problem but isn't something that I've strived for, it's just that so far
all resource managers have worked that way so that's how I think of it.
There is no technical reason for this to be true however so how about we
just say that in the future jobid's have to be alphanumeric strings,
this would work in this case although would have the downside you
couldn't specify the job as 27611 in the case above.

padb --show-jobs should show you what padb thinks the job id's are and
of course using -a rather than specifying a job tells it to use all jobs
so it'll just attempt to target one in the case above, regardless of
what it thinks it's called.

I'd be happy for a patch supporting either implementation, i.e. I don't
have a strong preference either way.   You can either have the jobid
encompass both the number and the server or you could continue with what
I attempted to encode in the patch I sent you where the job id is the
number and the server becomes a configuration option.

Actually this could make life easier for slurm and the way it handles
job steps, it effectively appends ".0" to the padb job id before handing
it over to slurm so this could probably be simplified if the .0 became a
optional part of the job id itself rather than a separate configuration
option.

> I am waiting for your patch (or reply) to continue.

I hope this helps you along the way, I can't really code anything from
here as I don't have access to a pbs system.

Ashley,

-- 

Ashley Pittman, Bath, UK.

Padb - A parallel job inspection tool for cluster computing
http://padb.pittman.org.uk



-------------- next part --------------
An HTML attachment was scrubbed...
URL: 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: new.patch.diff
Type: application/octet-stream
Size: 7143 bytes
Desc: not available
URL: 

From ashley at pittman.co.uk  Wed Nov 18 17:10:10 2009
From: ashley at pittman.co.uk (Ashley Pittman)
Date: Wed, 18 Nov 2009 17:10:10 +0000
Subject: [padb]
 =?iso-8859-1?q?R=E9f=2E_=3A_Re=3A_R=E9f=2E_=3A_Re=3A_=5Bpa?=
 =?iso-8859-1?q?db-devel=5D_Patch_for_Support_of_PBS_Pro_resource_manager?=
In-Reply-To: 
References: 
Message-ID: <1258564210.3385.23.camel@alpha>

On Wed, 2009-11-18 at 16:48 +0100, thipadin.seng-long at bull.net wrote:

> 1- Path of remote padb: 
> 
> [thipa at xn5 padb_open]$ ./padb -O rmgr=pbs -tx 27611 
> einner: xn20: bash: ./padb: No such file or directory 

> As the consequence path is not found, 
> So path to remote host must be a full path

I know of this one and don't have a generic solution, other resource
managers suffer from it as well, mpd springs to mind.  It should only
occur when developing padb as if you aren't running as ./ it's probably
installed somewhere and will also be installed on the remote nodes.

As a workaround I often type `pwd`/padb which causes it to work, it's
not ideal however.

> I did the patch as follows: 
>  [snip]

> If you have another idea i take it. 

How does this work if you do say ./src/padb -axt?  If it works in that
case then I'm happy with the code and I'll commit it, I've not added
anything before as I couldn't think of a generalised solution.

> 2- Use of uninitialized value in subtraction (-) at ./padb line 4077 
> 
>  
> 4077     foreach my $proc ( 0 .. $comm_data->{nprocesses} - 1 ) { 

Are you able to extract the process count from the job id and return it
as "nprocesses" in the hash returned by pbs_setup_job()?  I'm not
familiar with qstat so I don't know how to find this information.

> 3- Question about starting inner padb: 
> 
> How can I start an inner padb by hand on a remote host to debug such
> as: 
> perl -d ./padb --inner --jobid=27611.xn0 --stack-trace -O rmgr="pbs"
> --line-formatted 
> like I did it before, because this command doesn't work anymore. 
> You have changed it with "call back" and communication on ports. 

You're right in that debugging padb in the new model is a lot more
difficult, --debug full_duplex=all will show all comms  between the
inner and the outer process or use --debug all=all and padb will spit
out as much as it can.

I'm not familiar with perl -d so can't help you on that front.

> Here is the diff again r311 (diff r311 newone). 
> 
> So you can integrate my new patch and try to correct the point 2, 
> and send me back the new one, i will test it over. 

I'll be able to take a closer look when I'm back from SC, I only have my
netbook with me and aren't able to test anything from here, the patch
looks good so far however.

Ashley,

-- 

Ashley Pittman, Bath, UK.

Padb - A parallel job inspection tool for cluster computing
http://padb.pittman.org.uk




From thipadin.seng-long at bull.net  Fri Nov 20 15:33:37 2009
From: thipadin.seng-long at bull.net (thipadin.seng-long at bull.net)
Date: Fri, 20 Nov 2009 16:33:37 +0100
Subject: [padb] =?iso-8859-1?q?R=E9f=2E_=3A_Re=3A_R=E9f=2E_=3A_Re=3A_R=E9f?=
 =?iso-8859-1?q?=2E_=3A_Re=3A_=5B_padb-devel=5D_Patchfor_Support_of_PBS_Pr?=
 =?iso-8859-1?q?o_resource_manager?=
Message-ID: 

On 2009-11-18 at 06:10 PM , ashley at pittman.co.uk wrote:

> How does this work if you do say ./src/padb -axt?  If it works in that
> case then I'm happy with the code and I'll commit it, I've not added
> anything before as I couldn't think of a generalised solution.

I have a generalised solution now (in pbs_setup_job):
I have tested with:     ../padb
                        DirTest/padb
                        ./DirTest/padb
                        ./padb
                        /full_path/padb
                        padb (/usr/bin/padb)

>> 4077     foreach my $proc ( 0 .. $comm_data->{nprocesses} - 1 ) { 

> Are you able to extract the process count from the job id and return it
> as "nprocesses" in the hash returned by pbs_setup_job()?  I'm not
> familiar with qstat so I don't know how to find this information.

Ok it is done in pbs_get_lqsub when it treats "qstat -w -n -u $user 
\@$server" data
The $nprocess is stored in $pbs_tabjobs{$job}{nproc}, and will used in 
pbs_setup_job.

The Output test look like:

[thipa at xn5 padb_open]$ DirTest/padb -O rmgr=pbs -O stack-shows-locals=no 
-O stack-shows-params=no  -tx 27616
Warning, failed to locate ranks [5,7]
-----------------
[0-1,4,6] (4 processes)
-----------------
main() at pp_sndrcv_spbl.c:50
  PMPI_Finalize() at ?:?
    MPID_Finalize() at ?:?
      MPIDI_CH3_Progress_wait() at ?:?
        MPIDU_Sock_wait() at ?:?
          poll() at ?:?
-----------------
[2-3] (2 processes)
-----------------
ThreadId: 1
  -----------------
  [2] (1 processes)
  -----------------
  main() at pp_sndrcv_spbl.c:46
    PMPI_Recv() at ?:?
      MPID_Progress_wait() at ?:?
        MPIDI_CH3_Progress_wait() at ?:?
          MPIDU_Sock_wait() at ?:?
 
              ThreadId: 2
                start_thread() at ?:?
                  fd_server() at server.c:354
                    select() at ?:?
  -----------------
  [3] (1 processes)
  -----------------
  main() at pp_sndrcv_spbl.c:50
    PMPI_Finalize() at ?:?
      MPID_Finalize() at ?:?
        MPIDI_CH3_Progress_wait() at ?:?
          MPIDU_Sock_wait() at ?:?
            poll() at ?:?
              ThreadId: 2
                start_thread() at ?:?
                  fd_server() at server.c:354
                    select() at ?:?
[thipa at xn5 padb_open]$ 

Remark: Warning, failed to locate ranks [5,7]==> it is normal (processes 
terminated)

> I'll be able to take a closer look when I'm back from SC, I only have my
> netbook with me and aren't able to test anything from here, the patch
> looks good so far however.

I am giving you the new patch with diff -u here.



Hope you will look at it soon.
I think there is no more problem now and say it works.


Regards.
Thipadin.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: padb-pbs-new.patch
Type: application/octet-stream
Size: 9094 bytes
Desc: not available
URL: 

From padb at googlecode.com  Tue Nov 24 11:17:35 2009
From: padb at googlecode.com (padb at googlecode.com)
Date: Tue, 24 Nov 2009 11:17:35 +0000
Subject: [padb] r330 committed - Specify the full and correct path to padb
	when launching the inner...
Message-ID: <0016e64b0bc0d09fcc04791c1629@google.com>

Revision: 330
Author: apittman
Date: Tue Nov 24 03:17:18 2009
Log: Specify the full and correct path to padb when launching the inner
command instead of simply using $0.  Based on a patch from
Thipadin Seng-Long @ Bull.

http://code.google.com/p/padb/source/detail?r=330

Modified:
  /trunk/src/padb

=======================================
--- /trunk/src/padb	Thu Nov 12 07:14:50 2009
+++ /trunk/src/padb	Tue Nov 24 03:17:18 2009
@@ -4657,6 +4657,32 @@
      print "Failed to load secret from file ($file)\n";
      exit 1;
  }
+
+# Find the path to use for padb, previously we just used $0 here however
+# when using pdsh to find jobs $PWD isn't preserved between the inner and
+# the outer so we have to search out the real location of padb.  It would
+# be possible to do this on request, that is only for pdsh rather than
+# globally however there should be no down-side to always executing this.
+# Based on a patch from Thipadin Seng-Long @ Bull.
+
+sub padb_path {
+    my $pwd   = $ENV{PWD};
+    my $dirnm = dirname($0);
+
+    # if padb is launch as padb then dirname is .
+    # if padb is launched with a full path then dir is full
+    # if padb is launched as ../padb then dir is ..
+    # if padb is launched as Dir/padb then dir is Dir
+    my $out;
+    if ( $dirnm eq "." ) {    # started in current dir
+        my $base = basename($0);
+        return "$pwd\/$base";
+    } elsif ( $dirnm =~ m{\A/}x ) {    # started as full path or path known
+        return $0;
+    }
+
+    return "$pwd\/$0";
+}

  sub go_job {
      my $jobid = shift;
@@ -4692,7 +4718,7 @@
      debug_log( 'verbose', undef, 'There are %d processes over %d hosts',
          $ncpus, $nhosts );

-    $cmd .= " $0 --inner";
+    $cmd .= " " . padb_path() . " --inner";

      if ( $conf{inner_callback} ) {
          $secret = find_padb_secret();



From padb at googlecode.com  Tue Nov 24 11:36:53 2009
From: padb at googlecode.com (padb at googlecode.com)
Date: Tue, 24 Nov 2009 11:36:53 +0000
Subject: [padb] r331 committed - Add support for PBS pro. This patch was
	written by Thipadin Seng-Long...
Message-ID: <0016e64dde58d26c7104791c5ba4@google.com>

Revision: 331
Author: apittman
Date: Tue Nov 24 03:36:15 2009
Log: Add support for PBS pro.  This patch was written by Thipadin Seng-Long  
@ Bull.
Support jobs running under pbs where pbs is used both as a scheduler
and also as a resource manager (that is individual processes are launched
by the per-node pbs_attach deamon).

http://code.google.com/p/padb/source/detail?r=331

Modified:
  /trunk/src/padb

=======================================
--- /trunk/src/padb	Tue Nov 24 03:17:18 2009
+++ /trunk/src/padb	Tue Nov 24 03:36:15 2009
@@ -29,6 +29,7 @@
  # Revision history

  # Version 3.?
+#  * Support of PBS Pro
  #  * Add variables to tree based stack traces.
  #  * Solaris port.  Limited functionality compared to running on Linux
  #    however stack trace mode works fully.
@@ -336,7 +337,7 @@

  # Config options the inner knows about, only forward options if they are in
  # this list.
-my @inner_conf = qw(edb edbopt minfo rmgr scripts slurm_job_step);
+my @inner_conf = qw(edb edbopt minfo rmgr scripts slurm_job_step  
pbs_server);

  # More config options the inner knows about, these are forwarded on the
  # command line rather than over the sockets.
@@ -471,6 +472,13 @@
      inner_rmgr      => 'local',
  };

+$rmgr{pbs} = {
+    is_installed    => \&pbs_is_installed,
+    get_active_jobs => \&pbs_get_jobs,
+    setup_job       => \&pbs_setup_job,
+    find_pids       => \&pbs_find_pids,
+};
+
   
###############################################################################
  #
  # Config options
@@ -518,6 +526,8 @@

  $conf{slurm_job_step} = 0;

+$conf{pbs_server} = undef;
+
  # These settings are passed onto inner only.
  $conf{edbopt} = undef;

@@ -2621,6 +2631,115 @@

   
###############################################################################
  #
+# pbs support.
+#
+###############################################################################
+
+my %pbs_tabjobs;
+
+sub pbs_is_installed {
+    return find_exe("qstat");
+}
+
+# Load a list of jobs from a given server, saving the server and the host  
list
+# for each one.
+sub pbs_get_lqsub {
+    my ( $user, $server ) = @_;
+    my $job;
+    my $nprocess;
+    my $cmd = "qstat -w -n -u $user \@$server";
+
+    my @output = slurp_cmd($cmd);
+    foreach (@output) {
+        if (/\d+\.$server/i) {
+            $_ =~ s/^ +//;    # suppress leading space(for sure)
+            my @champs = split(/\s+/);    # split by space
+            if ( $champs[9] eq 'R' ) {    # take only Running
+                ($job) = split qr{\.}, $champs[0];
+                $nprocess = $champs[6];
+                push( @{ $pbs_tabjobs{$job}{nproc} }, $nprocess );
+            } else {
+                $job = undef;
+            }
+        } elsif ( defined $job ) {
+            $_ =~ s/^ +//;                # suppress blank in front of line
+            $_ =~ s/^\+//;                # suppress first + sign
+            my @champs = split(/\+/);     # split by '+'
+            if ( defined $pbs_tabjobs{$job}{server} ) {
+                printf("Warning, job $job exists on multiple servers\n");
+                next;
+            }
+            $pbs_tabjobs{$job}{server} = $server;
+            foreach my $word (@champs) {
+                chomp($word);
+                $word =~ s/\/.*//;        # take all from /
+                push( @{ $pbs_tabjobs{$job}{hosts} }, $word );
+            }
+        }
+    }
+}
+
+sub pbs_get_data {
+    my $user = shift;
+    return \%pbs_tabjobs if ( keys %pbs_tabjobs != 0 );
+
+    my @servers;
+
+    # Chose a list of servers to use, if one is specified by the user
+    # it will appear in $conf{pbs_server} here.  If one is not set
+    # load a list of available ones and use that.  It may be possible
+    # to have multiple jobs on different servers with the same jobid,
+    # if that is the case detect it in pbs_get_lqsub() and warn the
+    # user.  This will then force the user to expliciatly chose one
+    # of the servers.
+    if ( defined $conf{pbs_server} ) {
+        push @servers, $conf{pbs_server};
+    } else {
+        my @handle = slurp_cmd('qstat -fB');
+        foreach my $line (@handle) {
+            next if ( $line =~ /^\s+/ );    # skip if line begin with space
+            if ( $line =~ /Server:/ ) {
+                $line =~ s/^ +//;           # take off space at start
+                my @champs = split( /\s+/, $line );    # split buff by  
space
+                push( @servers, $champs[1] );
+            }
+        }
+    }
+
+    foreach my $server (@servers) {
+        pbs_get_lqsub( $user, $server );               # get job list by  
qsub
+    }
+    return \%pbs_tabjobs;
+}
+
+sub pbs_get_jobs {
+    my $user = shift;
+
+    my $d = pbs_get_data($user);
+
+    my @jobs = keys %{$d};
+    return @jobs;
+}
+
+sub pbs_setup_job {
+    my $job = shift;
+    my $d   = pbs_get_data($target_user);
+
+    my @hosts  = @{ $d->{$job}{hosts} };
+    my @nprocs = @{ $d->{$job}{nproc} };
+    my $nprocs = $nprocs[0];
+
+    config_set_internal( 'pbs_server', $d->{$job}{server} );
+
+    my %pcmd;
+
+    $pcmd{nprocesses} = $nprocs;
+    $pcmd{nhosts}     = @hosts;
+    @{ $pcmd{host_list} } = @hosts;
+
+    return %pcmd;
+}
+
  # open support.
  #
   
###############################################################################
@@ -6925,6 +7044,57 @@
      close $CMDS;
      return;
  }
+
+sub get_remote_env_bygdb {
+    my $pid = shift;
+
+    my %env;
+    my ( $fh, $filetmp ) = tempfile("/tmp/padb.XXXXXX");
+    print $fh 'set pagination off';
+    print $fh "\n";
+    print $fh 'set $envp = *(char ***) &__environ';
+    print $fh "\n";
+    print $fh 'while (*$envp != 0)';
+    print $fh "\n";
+    print $fh 'printf "%s\n",*$envp';
+    print $fh "\n";
+    print $fh 'set $envp = $envp + 1';
+    print $fh "\n";
+    print $fh 'end';
+    print $fh "\n";
+    close $fh;
+    my $psg = {
+        rdr => "",
+        wtr => "",
+        err => "",
+    };
+    my $ret_pid;
+    my $cmd = "gdb -nx -batch -x $filetmp -pid=";
+    $cmd .= $pid;
+    $ret_pid = open3( $psg->{wtr}, $psg->{rdr}, $psg->{err}, $cmd );
+    my $handle = $psg->{rdr};
+
+    while (<$handle>) {
+        next if (/^\[/);
+        next if (/^Using\s+/i);
+        next if (/^0x/i);
+        if (/=/) {
+            chomp;
+            my @f   = split "=";
+            my $key = $f[0];
+            if ( $f[1] !~ /^\(\)/ ) {    # not register function
+                shift @f;
+                $env{$key} = join( "=", @f );
+            }
+        }
+    }
+    close $psg->{wtr};
+    close $psg->{rdr};
+    close $psg->{err};
+    waitpid( $ret_pid, 0 );
+    unlink($filetmp);
+    return %env;
+}

  sub get_remote_env {
      my $pid = shift;
@@ -7913,9 +8083,10 @@
  }

  sub is_resmgr_process {
-    my $pid  = shift;
+    my $pid = shift;
      my $name = find_from_status( $pid, 'Name' );
-    my $mgrs = { rmsloader => 1, slurmd => 1, slurmstepd => 1 };
+    my $mgrs =
+      { rmsloader => 1, slurmd => 1, slurmstepd => 1, pbs_attach => 1 };
      return 1 if ( defined $mgrs->{$name} );
      return;
  }
@@ -7958,6 +8129,46 @@
      }
      return;
  }
+
+#
+# PBS support
+#
+sub pbs_find_pids {
+    my $job = shift;
+
+    if ( defined $inner_conf{pbs_server} ) {
+        $job .= ".$inner_conf{pbs_server}";
+    }
+
+    my %vps;
+
+    # Iterate over all processes for this user
+    foreach my $pid ( get_process_list($target_user) ) {
+
+        # Skip over resource manager processes.
+        next if ( is_resmgr_process($pid) );
+
+        # Skip over ones which aren't direct descendants of a resource  
manager
+        next unless is_parent_resmgr($pid);
+
+        my $vp;
+        my %env = get_remote_env($pid);
+        if ( !defined( $env{PBS_JOBID} ) || !defined( $env{PMI_RANK} ) ) {
+            %env = get_remote_env_bygdb($pid);
+        }
+
+        if ( $env{PBS_JOBID} eq $job ) {
+            $vp = $env{PMI_RANK};
+        }
+        if ( defined $vp ) {
+            $vps{$vp} = $pid;
+        }
+    }
+    foreach my $vp ( keys %vps ) {
+        my $pid = $vps{$vp};
+        maybe_show_pid( $vp, $pid );
+    }
+}

  sub rms_find_pids {
      my $jobid = shift;



From ashley at pittman.co.uk  Tue Nov 24 11:53:29 2009
From: ashley at pittman.co.uk (Ashley Pittman)
Date: Tue, 24 Nov 2009 11:53:29 +0000
Subject: [padb]
 =?iso-8859-1?q?R=E9f=2E_=3A_Re=3A_R=E9f=2E_=3A_Re=3A_R=E9f?=
 =?iso-8859-1?q?=2E_=3A_Re=3A_=5Bpadb-devel=5D_Patch_for_Support_of_PBS_Pr?=
 =?iso-8859-1?q?o_resource_manager?=
In-Reply-To: 
References: 
Message-ID: <1259063609.3358.24.camel@alpha>

On Fri, 2009-11-20 at 16:33 +0100, thipadin.seng-long at bull.net wrote:
> 
> On 2009-11-18 at 06:10 PM , ashley at pittman.co.uk wrote: 
> 
> > How does this work if you do say ./src/padb -axt?  If it works in
> that
> > case then I'm happy with the code and I'll commit it, I've not added
> > anything before as I couldn't think of a generalised solution. 
> 
> I have a generalised solution now (in pbs_setup_job): 

I've committed this, it's called in all cases for all resource managers
rather than just for PBS.

> Remark: Warning, failed to locate ranks [5,7]==> it is normal
> (processes terminated) 

You can set the configuration option "check_signon" to "none" to disable
this check if you wish.  If you have applications where this is
acceptable then you might like to write it in a configuration file to
make it the default for all users.

> I think there is no more problem now and say it works. 

Committed as r331 largely as you sent it.  I removed the pbs_port
variable as it was only ever read but not set.  Please give it a try and
let me know of any problems.

Ashley,

-- 

Ashley Pittman, Bath, UK.

Padb - A parallel job inspection tool for cluster computing
http://padb.pittman.org.uk




From thipadin.seng-long at bull.net  Wed Nov 25 10:15:01 2009
From: thipadin.seng-long at bull.net (thipadin.seng-long at bull.net)
Date: Wed, 25 Nov 2009 11:15:01 +0100
Subject: [padb] =?iso-8859-1?q?R=E9f=2E_=3A_Re=3A_R=E9f=2E_=3A_Re=3A_R=E9f?=
 =?iso-8859-1?q?=2E_=3A_Re=3A_R=E9f=2E_=3A_Re=3A_=5Bpadb-devel=5D_Patch_fo?=
 =?iso-8859-1?q?r_Support_of_PBS__Pro_resource_manager?=
Message-ID: 

Hi,

I have given a try for your last commit(r331) including so PBS support.
It seems to work now. You have done all the corrections (at your manner) I 
have sent.
Thanks you to accept to commit.
Now let's take a look to what is out from padb on a PBS job:

qstat -an:
xn0: 
                                                            Req'd  Req'd 
Elap
Job ID          Username Queue    Jobname    SessID NDS TSK Memory Time  S 
Time
--------------- -------- -------- ---------- ------ --- --- ------ ----- - 
-----
27617.xn0       thipa    workq    STDIN        1145   3   9    --    --  R 
00:15
   xn19/0*3+xn20/0*3+xn21/0*3
[thipa at xn5 padb_open]$

[thipa at xn5 padb_open]$ DirTest/padb -O rmgr=pbs -O stack-shows-locals=no 
-O stack-shows-params=no -O check-signon=none  -tx 27617
-----------------
[0,3,6] (3 processes)
-----------------
ThreadId: 1
  main() at pp_sndrcv_spbl.c:50
    PMPI_Finalize() at ?:?
      MPID_Finalize() at ?:?
        MPIDI_CH3_Progress_wait() at ?:?
          MPIDU_Sock_wait() at ?:?
            poll() at ?:?
              ThreadId: 2
                start_thread() at ?:?
                  fd_server() at server.c:354
                    select() at ?:?
-----------------
[1,4,7] (3 processes)
-----------------
main() at pp_sndrcv_spbl.c:50
  PMPI_Finalize() at ?:?
    MPID_Finalize() at ?:?
      MPIDI_CH3_Progress_wait() at ?:?
        MPIDU_Sock_wait() at ?:?
          poll() at ?:?
-----------------
[2] (1 processes)
-----------------
main() at pp_sndrcv_spbl.c:46
  PMPI_Recv() at ?:?
    MPID_Progress_wait() at ?:?
      MPIDI_CH3_Progress_wait() at ?:?
        MPIDU_Sock_wait() at ?:?
          poll() at ?:?
[thipa at xn5 padb_open]$ 


Next, I'll send you a version that supports slurm combined with openmpi 
mpirun (orte).
This case is used in our company. I mean it's not a pure slurm, neither a 
pure orte job.
It's a combination.
I will be working against r331.

Thank for every thing
Thipadin. Regards.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: 

From ashley at pittman.co.uk  Wed Nov 25 10:48:25 2009
From: ashley at pittman.co.uk (Ashley Pittman)
Date: Wed, 25 Nov 2009 10:48:25 +0000
Subject: [padb]
 =?iso-8859-1?q?R=E9f=2E_=3A_Re=3A_R=E9f=2E_=3A_Re=3A_R=E9f?=
 =?iso-8859-1?q?=2E_=3A_Re=3A_R=E9f=2E_=3A_Re=3A_=5Bpadb-devel=5D_Patch_fo?=
 =?iso-8859-1?q?r_Support_of_PBS_Pro_resource_manager?=
In-Reply-To: 
References: 
Message-ID: <1259146105.3358.1504.camel@alpha>

On Wed, 2009-11-25 at 11:15 +0100, thipadin.seng-long at bull.net wrote:
> I have given a try for your last commit(r331) including so PBS
> support. 
> It seems to work now. You have done all the corrections (at your
> manner) I have sent. 
> Thanks you to accept to commit. 
> Now let's take a look to what is out from padb on a PBS job: 

Good to see it's all working, thank you for the patch.  Are you able to
file a enhancement request against PBS Pro for a way of identifying
processes without digging into the environment?  When I spoke to them at
SC they didn't seem interested but then I don't have a support contract
with them.

> Next, I'll send you a version that supports slurm combined with
> openmpi mpirun (orte). 
> This case is used in our company. I mean it's not a pure slurm,
> neither a pure orte job. 

This is when using salloc with orterun to launch the job?  I'll look
forward to it.

Ashley,

-- 

Ashley Pittman, Bath, UK.

Padb - A parallel job inspection tool for cluster computing
http://padb.pittman.org.uk




From ashley at pittman.co.uk  Wed Nov 25 11:09:05 2009
From: ashley at pittman.co.uk (Ashley Pittman)
Date: Wed, 25 Nov 2009 11:09:05 +0000
Subject: [padb]
 =?iso-8859-1?q?R=E9f=2E_=3A_Re=3A_R=E9f=2E_=3A_Re=3A_R=E9f?=
 =?iso-8859-1?q?=2E_=3A_Re=3A_R=E9f=2E_=3A_Re=3A_=5Bpadb-devel=5D_Patch_fo?=
 =?iso-8859-1?q?r_Support_of_PBS_Pro_resource_manager?=
In-Reply-To: 
References: 
	
Message-ID: <1259147345.3358.1546.camel@alpha>

On Wed, 2009-11-25 at 11:37 +0100, Sylvain Jeaugey wrote:
> That's great news. For the future developments, we will need to support a 
> large number of combinations, and in this respect, I'm wondering if the 
> rmgr approach is fine enough. Maybe a two-steps approach would be better.
> 
> Here is how I would love to configure padb :
> jobmgr = slurm / pbs/ lsf / local / ...
> relay = none / orte / mpd / ...
> mpi = openmpi / mpich / mpich2
> 
> The jobmgr step would convert a jobid into a list of host/pid, the relay 
> step would follow childs (or not, e.g. for pure srun launch), and the mpi 
> step would get the MPI rank depending on the MPI library.
> 
> Maybe "relay" would be better defined as "launch system" to cope with e.g. 
> blaunch under lsf. I don't know the internals of padb, but maybe this 
> approach would remove the need to create a new rmgr each time a new 
> combination is used.
> 
> Ashley, what do you think about it ? Do you see how it could be done 
> inside padb ?

I've been thinking the very same thing, see this commit which documents
my thoughts.

http://code.google.com/p/padb/source/detail?r=329

Specifically to your comments the mpi step is really the same as the
jobmgr, some resource managers let you find the target pids on the front
end (orte/mpd/slurm) and some let you find them when on the destination
node (slurm/rms/mpd), even if both are provided it's not easy to pick
which one to use, slurm will tell you local pids only but mpd will tell
you all pids for all nodes.  Currently only orte and mpirun do the
pid=>rank conversion on the front end although mpd would benefit from it
as well.

There are really also (at least) three strata of resource managers,
there are the normal full-feature ones (orte/slurm/mpd/rms/pbs), a
middle layer which is the generic "mpirun" and finally the local ones,
local,local-fd and probably local-exe soon as well.

The launch system is the most critical aspect of scalability for large
jobs, it may be that I concentrate work on one single relay (probably
orte) for this case and insist that it's the only relay used for say 4k
or higher process count jobs.

Finally there is lsf-rms which is really rms but allows lsf job
identifiers to be specified and converted to rms identifiers
transparently by padb.

Ashley,

-- 

Ashley Pittman, Bath, UK.

Padb - A parallel job inspection tool for cluster computing
http://padb.pittman.org.uk




From sylvain.jeaugey at bull.net  Wed Nov 25 10:37:21 2009
From: sylvain.jeaugey at bull.net (Sylvain Jeaugey)
Date: Wed, 25 Nov 2009 11:37:21 +0100 (CET)
Subject: [padb]
	=?iso-8859-15?q?R=E9f=2E_=3A_Re=3A_R=E9f=2E_=3A_Re=3A_R=E9?=
	=?iso-8859-15?q?f=2E_=3A_Re=3A_R=E9f=2E_=3A_Re=3A_=5Bpadb-devel=5D?=
	=?iso-8859-15?q?_Patch_for_Support_of_PBS_Pro_resource_manager?=
In-Reply-To: 
References: 
Message-ID: 

Hi all.

That's great news. For the future developments, we will need to support a 
large number of combinations, and in this respect, I'm wondering if the 
rmgr approach is fine enough. Maybe a two-steps approach would be better.

Here is how I would love to configure padb :
jobmgr = slurm / pbs/ lsf / local / ...
relay = none / orte / mpd / ...
mpi = openmpi / mpich / mpich2

The jobmgr step would convert a jobid into a list of host/pid, the relay 
step would follow childs (or not, e.g. for pure srun launch), and the mpi 
step would get the MPI rank depending on the MPI library.

Maybe "relay" would be better defined as "launch system" to cope with e.g. 
blaunch under lsf. I don't know the internals of padb, but maybe this 
approach would remove the need to create a new rmgr each time a new 
combination is used.

Ashley, what do you think about it ? Do you see how it could be done 
inside padb ?

Sylvain

On Wed, 25 Nov 2009, thipadin.seng-long at bull.net wrote:

> 
> Hi,
> 
> I have given a try for your last commit(r331) including so PBS support.
> It seems to work now. You have done all the corrections (at your manner) I have sent.
> Thanks you to accept to commit.
> Now let's take a look to what is out from padb on a PBS job:
> 
> qstat -an:
> xn0:
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Req'd ?Req'd ? Elap
> Job ID ? ? ? ? ?Username Queue ? ?Jobname ? ?SessID NDS TSK Memory Time ?S Time
> --------------- -------- -------- ---------- ------ --- --- ------ ----- - -----
> 27617.xn0 ? ? ? thipa ? ?workq ? ?STDIN ? ? ? ?1145 ? 3 ? 9 ? ?-- ? ?-- ?R 00:15
> ? ?xn19/0*3+xn20/0*3+xn21/0*3
> [thipa at xn5 padb_open]$
> 
> [thipa at xn5 padb_open]$ DirTest/padb -O rmgr=pbs -O stack-shows-locals=no ?-O stack-shows-params=no -O check-signon=none
> ?-tx 27617
> -----------------
> [0,3,6] (3 processes)
> -----------------
> ThreadId: 1
> ? main() at pp_sndrcv_spbl.c:50
> ? ? PMPI_Finalize() at ?:?
> ? ? ? MPID_Finalize() at ?:?
> ? ? ? ? MPIDI_CH3_Progress_wait() at ?:?
> ? ? ? ? ? MPIDU_Sock_wait() at ?:?
> ? ? ? ? ? ? poll() at ?:?
> ? ? ? ? ? ? ? ThreadId: 2
> ? ? ? ? ? ? ? ? start_thread() at ?:?
> ? ? ? ? ? ? ? ? ? fd_server() at server.c:354
> ? ? ? ? ? ? ? ? ? ? select() at ?:?
> -----------------
> [1,4,7] (3 processes)
> -----------------
> main() at pp_sndrcv_spbl.c:50
> ? PMPI_Finalize() at ?:?
> ? ? MPID_Finalize() at ?:?
> ? ? ? MPIDI_CH3_Progress_wait() at ?:?
> ? ? ? ? MPIDU_Sock_wait() at ?:?
> ? ? ? ? ? poll() at ?:?
> -----------------
> [2] (1 processes)
> -----------------
> main() at pp_sndrcv_spbl.c:46
> ? PMPI_Recv() at ?:?
> ? ? MPID_Progress_wait() at ?:?
> ? ? ? MPIDI_CH3_Progress_wait() at ?:?
> ? ? ? ? MPIDU_Sock_wait() at ?:?
> ? ? ? ? ? poll() at ?:?
> [thipa at xn5 padb_open]$
> 
> 
> Next, I'll send you a version that supports slurm combined with openmpi mpirun (orte).
> This case is used in our company. I mean it's not a pure slurm, neither a pure orte job.
> It's a combination.
> I will be working against r331.
> 
> Thank for every thing
> Thipadin. Regards.
>

From padb at googlecode.com  Wed Nov 25 13:16:48 2009
From: padb at googlecode.com (padb at googlecode.com)
Date: Wed, 25 Nov 2009 13:16:48 +0000
Subject: [padb] r332 committed - Migrate orte from using setup_pcmd to using
	setup_job for setting up t...
Message-ID: <0016e649cd8400433d047931dfae@google.com>

Revision: 332
Author: apittman
Date: Wed Nov 25 05:15:57 2009
Log: Migrate orte from using setup_pcmd to using setup_job for setting up  
the
runtime to attach to a job.  Whilst I'm at it add a cleanup_cb and  
cleanup_handle
to the hash to replace the cleanup_pcmd function.

http://code.google.com/p/padb/source/detail?r=332

Modified:
  /trunk/src/padb

=======================================
--- /trunk/src/padb	Tue Nov 24 03:36:15 2009
+++ /trunk/src/padb	Wed Nov 25 05:15:57 2009
@@ -430,8 +430,7 @@
  $rmgr{orte} = {
      is_installed    => \&open_is_installed,
      get_active_jobs => \&open_get_jobs,
-    setup_pcmd      => \&open_setup_pcmd,
-    cleanup_pcmd    => \&open_cleanup_pcmd,
+    setup_job       => \&open_setup_job,
  };

  $rmgr{'lsf-rms'} = {
@@ -2805,9 +2804,7 @@
      return keys %open_jobs;
  }

-my $open_dfile;
-
-sub open_setup_pcmd {
+sub open_setup_job {
      my $job = shift;

      open_get_data();
@@ -2822,17 +2819,25 @@
      }
      close $fh;

-    $open_dfile = $fn;
-
      my $prefix = find_ompi_prefix();
      my $cmd    = "orterun -machinefile $fn -np $i $prefix";
-    my $hosts  = @hosts;
-
-    return ( $cmd, $open_jobs{$job}{nprocs}, $hosts,  
$open_jobs{$job}{ranks} );
+
+    my %pcmd;
+    $pcmd{nprocesses}   = $open_jobs{$job}{nprocs};
+    $pcmd{nhosts}       = @hosts;
+    $pcmd{process_data} = $open_jobs{$job}{ranks};
+    $pcmd{command}      = $cmd;
+    @{ $pcmd{host_list} } = @hosts;
+    $pcmd{cleanup_cb}     = \&open_cleanup_job;
+    $pcmd{cleanup_handle} = $fn;
+
+    return %pcmd;
+
  }

-sub open_cleanup_pcmd {
-    unlink $open_dfile if ( defined $open_dfile );
+sub open_cleanup_job {
+    my $file = shift;
+    unlink $file;
      return;
  }

@@ -4858,7 +4863,12 @@

      debug_log( 'verbose', undef, 'Completed command' );

+    if ( defined $pcmd{cleanup_cb} ) {
+        $pcmd{cleanup_cb}( $pcmd{cleanup_handle} );
+    }
+
      cleanup_pcmd();
+
      return $errors;
  }




From padb at googlecode.com  Wed Nov 25 13:50:10 2009
From: padb at googlecode.com (padb at googlecode.com)
Date: Wed, 25 Nov 2009 13:50:10 +0000
Subject: [padb] r333 committed - Convert mpd to use setup_job rather than
	setup_pcmd....
Message-ID: <0016e64c2536563e6704793256bd@google.com>

Revision: 333
Author: apittman
Date: Wed Nov 25 05:49:12 2009
Log: Convert mpd to use setup_job rather than setup_pcmd.
Handle the removal of the temory file with a callback function
(which is shared with open_setup_job) which allows me to get
rid of cleanup_pcmd() completely.
Return the pid=>rank mapping from mpd_setup_job which avoids
having to call mpdlistjobs on the target nodes.  As this was
calling a global operation from every node simultanously this
should help the startup time dramatically on larger mpich2 jobs.

http://code.google.com/p/padb/source/detail?r=333

Modified:
  /trunk/src/padb

=======================================
--- /trunk/src/padb	Wed Nov 25 05:15:57 2009
+++ /trunk/src/padb	Wed Nov 25 05:49:12 2009
@@ -359,7 +359,6 @@
  # get_active_jobs user List      yes      Return list of all active job  
for user.
  # is_job_running  job  Bool      no       Check if a given job is running.
  # job_to_key      job  key       no       Convert from jobId to shm key.
-# cleanup_pcmd    -    -         no       Cleans up and temporary files.
  # find_pids       job  -         maybe    Called on the inner to locate  
pids.

  # In addition one of these two is preferred setup_job has more
@@ -421,9 +420,7 @@
  $rmgr{mpd} = {
      is_installed           => \&mpd_is_installed,
      get_active_jobs        => \&mpd_get_jobs,
-    setup_pcmd             => \&mpd_setup_pcmd,
-    cleanup_pcmd           => \&mpd_cleanup_pcmd,
-    find_pids              => \&mpd_find_pids,
+    setup_job              => \&mpd_setup_job,
      require_inner_callback => 1,
  };

@@ -2574,10 +2571,9 @@
          }
          if ( $key eq 'pid' ) {
              $pid = $value;
-            $jobs{$job}{pids}{$host}{$value}++;
          }
          if ( $key eq 'rank' ) {
-            $jobs{$job}{pids}{$host}{$pid} = $value;
+            $jobs{$job}{pids}{$host}{$value} = $pid;
              if (   ( not defined $jobs{$job}{lastproc} )
                  or ( $value > $jobs{$job}{lastproc} ) )
              {
@@ -2598,9 +2594,7 @@
      return @jobs;
  }

-my $mpd_dfile;
-
-sub mpd_setup_pcmd {
+sub mpd_setup_job {
      my $job = shift;

      my $d = mpd_get_data();
@@ -2614,17 +2608,23 @@
      }
      close $fh;

-    $mpd_dfile = $fn;
-
      my $cmd = "mpdrun -machinefile $fn -np $i";

-    my $hosts = @hosts;
-
-    return ( $cmd, $d->{$job}{lastproc} + 1, $hosts );
+    my %pcmd;
+    $pcmd{nprocesses}   = $d->{$job}{lastproc} + 1;
+    $pcmd{nhosts}       = @hosts;
+    $pcmd{process_data} = $d->{$job}{pids};
+    $pcmd{command}      = $cmd;
+    @{ $pcmd{host_list} } = @hosts;
+    $pcmd{cleanup_cb}     = \&unlink_file;
+    $pcmd{cleanup_handle} = $fn;
+
+    return %pcmd;
  }

-sub mpd_cleanup_pcmd {
-    unlink $mpd_dfile if ( defined $mpd_dfile );
+sub unlink_file {
+    my $file = shift;
+    unlink $file;
      return;
  }

@@ -2828,18 +2828,12 @@
      $pcmd{process_data} = $open_jobs{$job}{ranks};
      $pcmd{command}      = $cmd;
      @{ $pcmd{host_list} } = @hosts;
-    $pcmd{cleanup_cb}     = \&open_cleanup_job;
+    $pcmd{cleanup_cb}     = \&unlink_file;
      $pcmd{cleanup_handle} = $fn;

      return %pcmd;

  }
-
-sub open_cleanup_job {
-    my $file = shift;
-    unlink $file;
-    return;
-}

   
###############################################################################
  #
@@ -2992,15 +2986,6 @@
      @{ $pcmd{host_list} } = @hosts;

      return %pcmd;
-
-    my $cmd = $EMPTY_STRING;
-    if ( $hosts[0] ne hostname() or @hosts > 1 ) {
-        my $hlist = join q{,}, @hosts;
-        $cmd = "pdsh -w $hlist";
-    }
-
-    my $hc = @hosts;
-    return ( $cmd, $nprocs, $hc, \%pt );
  }

   
###############################################################################
@@ -3177,11 +3162,13 @@
          my ( $cmd, $nprocesses, $nhosts, $pd ) =
            $rmgr{ $conf{rmgr} }{setup_pcmd}($job);
          my %pcmd = (
-            command      => $cmd,
-            nprocesses   => $nprocesses,
-            nhosts       => $nhosts,
-            process_data => $pd
+            command    => $cmd,
+            nprocesses => $nprocesses,
+            nhosts     => $nhosts,
          );
+        if ( defined $pd ) {
+            $pcmd{process_data} = $pd;
+        }
          return %pcmd;
      }

@@ -3213,14 +3200,6 @@
      }
      return %pcmd;
  }
-
-sub cleanup_pcmd {
-    my $job = shift;
-    if ( defined $rmgr{ $conf{rmgr} }{cleanup_pcmd} ) {
-        $rmgr{ $conf{rmgr} }{cleanup_pcmd}();
-    }
-    return;
-}

   
###############################################################################
  #
@@ -4866,8 +4845,6 @@
      if ( defined $pcmd{cleanup_cb} ) {
          $pcmd{cleanup_cb}( $pcmd{cleanup_handle} );
      }
-
-    cleanup_pcmd();

      return $errors;
  }
@@ -8127,18 +8104,6 @@
      maybe_show_pid( 0, $pid );
      return;
  }
-
-sub mpd_find_pids {
-    my $job = shift;
-    my $d   = mpd_get_data();
-
-    my $j = $d->{$job}{pids}{ $inner_conf{hostname} };
-
-    foreach my $pid ( keys %{$j} ) {
-        maybe_show_pid( $j->{$pid}, $pid );
-    }
-    return;
-}

  #
  # PBS support



From padb at googlecode.com  Wed Nov 25 14:01:13 2009
From: padb at googlecode.com (padb at googlecode.com)
Date: Wed, 25 Nov 2009 14:01:13 +0000
Subject: [padb] r334 committed - Migrate the 'local' sudo-resource-manager
	over from setup_pcmd...
Message-ID: <000e0cd20a2cd1b74f0479327d20@google.com>

Revision: 334
Author: apittman
Date: Wed Nov 25 06:00:09 2009
Log: Migrate the 'local' sudo-resource-manager over from setup_pcmd
to setup_job.

http://code.google.com/p/padb/source/detail?r=334

Modified:
  /trunk/src/padb

=======================================
--- /trunk/src/padb	Wed Nov 25 05:49:12 2009
+++ /trunk/src/padb	Wed Nov 25 06:00:09 2009
@@ -449,22 +449,21 @@
  $rmgr{local} = {
      get_active_jobs => \&local_get_jobs,
      job_is_running  => \&local_job_is_running,
-    setup_pcmd      => \&local_setup_pcmd,
-    find_pids       => \&local_find_pids,
+    setup_job       => \&local_setup_job,
  };

  $rmgr{'local-qsnet'} = {
      is_installed    => \&local_q_is_installed,
      get_active_jobs => \&local_q_get_jobs,
      job_is_running  => \&local_job_is_running,
-    setup_pcmd      => \&local_setup_pcmd,
+    setup_job       => \&local_setup_job,
      inner_rmgr      => 'local',
  };

  $rmgr{'local-fd'} = {
      get_active_jobs => \&local_fd_get_jobs,
      job_is_running  => \&local_job_is_running,
-    setup_pcmd      => \&local_setup_pcmd,
+    setup_job       => \&local_setup_job,
      inner_rmgr      => 'local',
  };

@@ -2533,8 +2532,16 @@
      return ( -d "/proc/$job" );
  }

-sub local_setup_pcmd {
-    return ( $EMPTY_STRING, 1, 1 );
+sub local_setup_job {
+    my $job = shift;
+
+    my $hostname = hostname();
+    my %pcmd;
+    $pcmd{nprocesses}                 = 1;
+    $pcmd{nhosts}                     = 1;
+    $pcmd{process_data}{$hostname}{0} = $job;
+    @{ $pcmd{host_list} } = $hostname;
+    return %pcmd;
  }

   
###############################################################################
@@ -8095,15 +8102,6 @@
      }
      return;
  }
-
-# Local processes per node, i.e. no resource manager support, we only have
-# one process in this case so call it process 0.
-sub local_find_pids {
-    my $pid = shift;
-
-    maybe_show_pid( 0, $pid );
-    return;
-}

  #
  # PBS support



From padb at googlecode.com  Wed Nov 25 14:22:26 2009
From: padb at googlecode.com (padb at googlecode.com)
Date: Wed, 25 Nov 2009 14:22:26 +0000
Subject: [padb] r335 committed - Rename vp to be rank in the proc-summary
	view, vp really is a Quadrics...
Message-ID: <0016e64eeca6b7350f047932c9d8@google.com>

Revision: 335
Author: apittman
Date: Wed Nov 25 06:21:06 2009
Log: Rename vp to be rank in the proc-summary view, vp really is a Quadrics  
term,
most of the rest of the industry use rank and changing to this will make it
easier for people to understand.

http://code.google.com/p/padb/source/detail?r=335

Modified:
  /trunk/src/padb

=======================================
--- /trunk/src/padb	Wed Nov 25 06:00:09 2009
+++ /trunk/src/padb	Wed Nov 25 06:21:06 2009
@@ -3386,7 +3386,7 @@
      my $lines = tree_from_namespace( $nlines->{target_data} );
      foreach my $tag ( keys %{$lines} ) {
          my %hash;
-        $hash{vp} = $tag;
+        $hash{rank} = $tag;
          foreach my $key ( keys %{ $lines->{$tag} } ) {

              my $value = $lines->{$tag}{$key};
@@ -8911,7 +8911,6 @@
          handler     => \&show_queue,
          help        => 'Show the message queues',
          options_i   => { mpi_dll => undef, }
-
      };

      $allfns{kill} = {
@@ -8947,7 +8946,6 @@
              show_group_members => 'no',
              show_all_groups    => 'no',
          },
-
      };

      $allfns{pinfo} = {
@@ -8959,8 +8957,7 @@
              proc_shows_fds  => 'no',
              proc_shows_maps => 'no',
              proc_shows_stat => 'no',
-          }
-
+        }
      };

      $allfns{proc_summary} = {
@@ -8970,7 +8967,7 @@
          help        => 'Show process information in top format',
          options_i   => {
              column_seperator => '  ',
-            proc_sort_key    => 'vp',
+            proc_sort_key    => 'rank',
              nprocs_output    => undef,
          },
          options_bool => {
@@ -8980,17 +8977,15 @@
              proc_shows_maps    => 'no',
              proc_show_header   => 'yes',
              reverse_sort_order => 'no',
-
          },
          secondary => [
              {
                  arg_long => 'proc_format',
                  type     => '=s',
                  default =>
-'vp=vpid,hostname,pid,vmsize,vmrss,stat.state=S,load1=uptime,pcpu=%cpu,stat.processor=lcore,name=command'
+'rank,hostname,pid,vmsize,vmrss,stat.state=S,load1=uptime,pcpu=%cpu,stat.processor=lcore,name=command'
              },
-          ]
-
+        ]
      };

      $allfns{stack} = {



From padb at googlecode.com  Wed Nov 25 17:37:41 2009
From: padb at googlecode.com (padb at googlecode.com)
Date: Wed, 25 Nov 2009 17:37:41 +0000
Subject: [padb] r336 committed - Move some code around and add error
	checking to show_proc_all such tha...
Message-ID: <0016e649cd8402ddff0479358424@google.com>

Revision: 336
Author: apittman
Date: Wed Nov 25 09:36:55 2009
Log: Move some code around and add error checking to show_proc_all such that
if running proc-summary in watch mode and the target job dies padb doesn't
spew errors at the user.  The outer process correctly reports that the
job has died so all this code needs to do is to not generate perl errors.

http://code.google.com/p/padb/source/detail?r=336

Modified:
  /trunk/src/padb

=======================================
--- /trunk/src/padb	Wed Nov 25 06:21:06 2009
+++ /trunk/src/padb	Wed Nov 25 09:36:55 2009
@@ -7345,7 +7345,10 @@
      if ( $carg->{proc_shows_proc} ) {
          foreach my $proc ( @{$list} ) {
              my $pid = $proc->{pid};
-            open $proc->{handle}, '<', "/proc/$pid/stat";
+            my $handle;
+            if ( open $handle, '<', "/proc/$pid/stat" ) {
+                $proc->{handle} = $handle;
+            }
          }

          open $SFD, '<', '/proc/stat';
@@ -7354,8 +7357,8 @@
          my $stat = <$SFD>;

          foreach my $proc ( @{$list} ) {
-            my $pid = $proc->{pid};
-            my $h   = $proc->{handle};
+            next unless ( defined $proc->{handle} );
+            my $h = $proc->{handle};
              $proc->{stat_start} = <$h>;
              seek $proc->{handle}, 0, 0;
          }
@@ -7386,10 +7389,15 @@
          my $stat = <$SFD>;

          foreach my $proc ( @{$list} ) {
-            my $pid = $proc->{pid};
-            my $h   = $proc->{handle};
+            next unless ( defined $proc->{handle} );
+            my $h = $proc->{handle};
              $proc->{stat_end} = <$h>;
              close $proc->{handle};
+	    # Set handle to undef to prepare for future iterations of the
+	    # loop if being run in watch mode.  Without this handle remains
+	    # set but refers to the closed fd which causes errors next time
+	    # round the loop if the open fails.
+	    $proc->{handle} = undef;
          }

          seek $SFD, 0, 0;
@@ -7414,6 +7422,12 @@
          foreach my $proc ( @{$list} ) {
              my $vp = $proc->{vp};

+            proc_output( $vp, 'load1',  $l1 );
+            proc_output( $vp, 'load5',  $l15 );
+            proc_output( $vp, 'load15', $l15 );
+
+            next unless defined $proc->{stat_end};
+
              proc_output(
                  $vp, 'pcpu',
                  pcpu_total(
@@ -7435,9 +7449,6 @@
                      $proc->{stat_start}, $proc->{stat_end}
                  )
              );
-            proc_output( $vp, 'load1',  $l1 );
-            proc_output( $vp, 'load5',  $l15 );
-            proc_output( $vp, 'load15', $l15 );
          }
      }




From padb at googlecode.com  Wed Nov 25 17:50:52 2009
From: padb at googlecode.com (padb at googlecode.com)
Date: Wed, 25 Nov 2009 17:50:52 +0000
Subject: [padb] r337 committed - When running in proc-summary mode only
	pause for a second if the perce...
Message-ID: <0016e648fca61c0fbc047935b3e0@google.com>

Revision: 337
Author: apittman
Date: Wed Nov 25 09:50:16 2009
Log: When running in proc-summary mode only pause for a second if the  
percent cpu
figures are going to be displayed to the user.

http://code.google.com/p/padb/source/detail?r=337

Modified:
  /trunk/src/padb

=======================================
--- /trunk/src/padb	Wed Nov 25 09:36:55 2009
+++ /trunk/src/padb	Wed Nov 25 09:50:16 2009
@@ -7381,7 +7381,24 @@
      }

      if ( $carg->{proc_shows_proc} ) {
-        sleep 1;
+
+        # We only need to sleep if we are sampling the jiffy count for
+        # this process, otherwise there is no need and it shows padb down
+        # considerably.
+        my $should_sleep = 1;
+        if ( defined $carg->{proc_format} ) {
+            $should_sleep = 0;
+            if (   defined $proc_keys{pcpu}
+                or defined $proc_keys{pucpu}
+                or defined $proc_keys{pscpu} )
+            {
+                $should_sleep = 1;
+            }
+        }
+
+        if ($should_sleep) {
+            sleep 1;
+        }

          seek $SFD, 0, 0;

@@ -7393,11 +7410,12 @@
              my $h = $proc->{handle};
              $proc->{stat_end} = <$h>;
              close $proc->{handle};
-	    # Set handle to undef to prepare for future iterations of the
-	    # loop if being run in watch mode.  Without this handle remains
-	    # set but refers to the closed fd which causes errors next time
-	    # round the loop if the open fails.
-	    $proc->{handle} = undef;
+
+            # Set handle to undef to prepare for future iterations of the
+            # loop if being run in watch mode.  Without this handle remains
+            # set but refers to the closed fd which causes errors next time
+            # round the loop if the open fails.
+            $proc->{handle} = undef;
          }

          seek $SFD, 0, 0;



From padb at googlecode.com  Wed Nov 25 18:00:10 2009
From: padb at googlecode.com (padb at googlecode.com)
Date: Wed, 25 Nov 2009 18:00:10 +0000
Subject: [padb] r338 committed - Run through perlcritic and get rid of the
	easy offenders from the new ...
Message-ID: <0016e64b9a1c6a9391047935d4f3@google.com>

Revision: 338
Author: apittman
Date: Wed Nov 25 09:59:00 2009
Log: Run through perlcritic and get rid of the easy offenders from the new  
pbs code.

http://code.google.com/p/padb/source/detail?r=338

Modified:
  /trunk/src/padb

=======================================
--- /trunk/src/padb	Wed Nov 25 09:50:16 2009
+++ /trunk/src/padb	Wed Nov 25 09:59:00 2009
@@ -2644,7 +2644,7 @@
  my %pbs_tabjobs;

  sub pbs_is_installed {
-    return find_exe("qstat");
+    return find_exe('qstat');
  }

  # Load a list of jobs from a given server, saving the server and the host  
list
@@ -2683,6 +2683,7 @@
              }
          }
      }
+    return;
  }

  sub pbs_get_data {



From padb at googlecode.com  Wed Nov 25 18:22:03 2009
From: padb at googlecode.com (padb at googlecode.com)
Date: Wed, 25 Nov 2009 18:22:03 +0000
Subject: [padb] r339 committed - Add missing return calls to the end of
	three more functions.
Message-ID: <0016e64be580a612db04793622b4@google.com>

Revision: 339
Author: apittman
Date: Wed Nov 25 10:21:40 2009
Log: Add missing return calls to the end of three more functions.

http://code.google.com/p/padb/source/detail?r=339

Modified:
  /trunk/src/padb

=======================================
--- /trunk/src/padb	Wed Nov 25 09:59:00 2009
+++ /trunk/src/padb	Wed Nov 25 10:21:40 2009
@@ -3621,7 +3621,7 @@
              }
          }
      }
-
+    return;
  }

  sub _add_data_to_tree {
@@ -5508,7 +5508,7 @@
      } else {
          _gdb_send_real( $gdb, '-gdb-set print address off' );
      }
-
+    return;
  }

  sub gdb_n_send {
@@ -8171,6 +8171,7 @@
          my $pid = $vps{$vp};
          maybe_show_pid( $vp, $pid );
      }
+    return;
  }

  sub rms_find_pids {



From ashley at pittman.co.uk  Wed Nov 25 18:51:23 2009
From: ashley at pittman.co.uk (Ashley Pittman)
Date: Wed, 25 Nov 2009 18:51:23 +0000
Subject: [padb]
 =?iso-8859-1?q?R=E9f=2E_=3A_Re=3A_R=E9f=2E_=3A_Re=3A_R=E9f?=
 =?iso-8859-1?q?=2E_=3A_Re=3A_R=E9f=2E_=3A_Re=3A_=5Bpadb-devel=5D_Patch_fo?=
 =?iso-8859-1?q?r_Support_of_PBS_Pro_resource_manager?=
In-Reply-To: 
References: 
Message-ID: <1259175083.3358.1966.camel@alpha>

On Wed, 2009-11-25 at 11:15 +0100, thipadin.seng-long at bull.net wrote:
> 
> Thanks you to accept to commit. 

How does the following look to you?  nproc is only ever set once so it
can be stored as a value rather than a array with only one entry which
simplifies the code slightly.  I would just commit it from here but
without access to a pbs system I can't test it.

Re: coding standards take a look at the 'tidy' and 'report' make
targets, I try and ensure my code has as few errors as reported by
perlcritic as possible.  This isn't always easy but is worth doing now
and again.  I also try and avoid implicitly using $_ where-ever possible
as it tends to make the code easier to read.

ashley at alpha:~/code/padb/trunk/src$ svn diff
Index: padb
===================================================================
--- padb	(revision 338)
+++ padb	(working copy)
@@ -2662,8 +2662,7 @@
             my @champs = split(/\s+/);    # split by space
             if ( $champs[9] eq 'R' ) {    # take only Running
                 ($job) = split qr{\.}, $champs[0];
-                $nprocess = $champs[6];
-                push( @{ $pbs_tabjobs{$job}{nproc} }, $nprocess );
+                $pbs_tabjobs{$job}{nproc} = $champs[6];
             } else {
                 $job = undef;
             }
@@ -2732,15 +2731,13 @@
     my $job = shift;
     my $d   = pbs_get_data($target_user);
 
-    my @hosts  = @{ $d->{$job}{hosts} };
-    my @nprocs = @{ $d->{$job}{nproc} };
-    my $nprocs = $nprocs[0];
+    my @hosts = @{ $d->{$job}{hosts} };
 
     config_set_internal( 'pbs_server', $d->{$job}{server} );
 
     my %pcmd;
 
-    $pcmd{nprocesses} = $nprocs;
+    $pcmd{nprocesses} = $d->{$job}{nproc};
     $pcmd{nhosts}     = @hosts;
     @{ $pcmd{host_list} } = @hosts;
 


-- 

Ashley Pittman, Bath, UK.

Padb - A parallel job inspection tool for cluster computing
http://padb.pittman.org.uk




From padb at googlecode.com  Wed Nov 25 20:58:10 2009
From: padb at googlecode.com (padb at googlecode.com)
Date: Wed, 25 Nov 2009 20:58:10 +0000
Subject: [padb] r340 committed - Add a couple of missing newlines to error
	messages in the mpirun...
Message-ID: <0016e64c2536f21c19047938509a@google.com>

Revision: 340
Author: apittman
Date: Wed Nov 25 12:57:47 2009
Log: Add a couple of missing newlines to error messages in the mpirun
resource manager.

http://code.google.com/p/padb/source/detail?r=340

Modified:
  /trunk/src/padb

=======================================
--- /trunk/src/padb	Wed Nov 25 10:21:40 2009
+++ /trunk/src/padb	Wed Nov 25 12:57:47 2009
@@ -2960,7 +2960,7 @@

      my $nprocs = gdb_read_value( $gdb, 'MPIR_proctable_size' );
      if ( not defined $nprocs ) {
-        print "No MPIR_proctable_size symbol found, cannot continue";
+        print "No MPIR_proctable_size symbol found, cannot continue\n";
          return;
      }

@@ -2983,7 +2983,7 @@
      my @hosts = keys(%pt);

      if ( @hosts == 0 ) {
-        print "No process data found";
+        print "No process data found\n";
          return;
      }




From thipadin.seng-long at bull.net  Thu Nov 26 09:27:16 2009
From: thipadin.seng-long at bull.net (thipadin.seng-long at bull.net)
Date: Thu, 26 Nov 2009 10:27:16 +0100
Subject: [padb] =?iso-8859-1?q?R=E9f=2E_=3A_Re=3A_R=E9f=2E_=3A_Re=3A_R=E9f?=
 =?iso-8859-1?q?=2E_=3A_Re=3A_R=E9f=2E_=3A_Re=3A_R=E9f=2E_=3A_Re=3A_=5Bpad?=
 =?iso-8859-1?q?b-devel=5D_Patch__for_Support_of_PBS_Pro_resource_manager?=
Message-ID: 

Hi,
I'm agree with what you said below.
Nproc is just one value not a list, it should be stored as it is.
My coding was just to follow the same method as for 'hosts' which is a 
list.
I have given a try with the new patch on the Pbs system. It's OK.

More later.
Thipadin.





Ashley Pittman 
11/25/2009 07:51 PM

 
        Pour :  thipadin.seng-long at bull.net
        cc :    florence.vallee at bull.net, francois.wellenreiter at bull.net, 
padb-devel at pittman.org.uk, Sylvain.JEAUGEY at bull.net
        Objet : Re: R?f. : Re: R?f. : Re: R?f. : Re: R?f. : Re: [padb-devel] Patch for 
Support of PBS Pro resource manager

On Wed, 2009-11-25 at 11:15 +0100, thipadin.seng-long at bull.net wrote:
> 
> Thanks you to accept to commit. 

How does the following look to you?  nproc is only ever set once so it
can be stored as a value rather than a array with only one entry which
simplifies the code slightly.  I would just commit it from here but
without access to a pbs system I can't test it.

Re: coding standards take a look at the 'tidy' and 'report' make
targets, I try and ensure my code has as few errors as reported by
perlcritic as possible.  This isn't always easy but is worth doing now
and again.  I also try and avoid implicitly using $_ where-ever possible
as it tends to make the code easier to read.

ashley at alpha:~/code/padb/trunk/src$ svn diff
Index: padb
===================================================================
--- padb                 (revision 338)
+++ padb                 (working copy)
@@ -2662,8 +2662,7 @@
             my @champs = split(/\s+/);    # split by space
             if ( $champs[9] eq 'R' ) {    # take only Running
                 ($job) = split qr{\.}, $champs[0];
-                $nprocess = $champs[6];
-                push( @{ $pbs_tabjobs{$job}{nproc} }, $nprocess );
+                $pbs_tabjobs{$job}{nproc} = $champs[6];
             } else {
                 $job = undef;
             }
@@ -2732,15 +2731,13 @@
     my $job = shift;
     my $d   = pbs_get_data($target_user);
 
-    my @hosts  = @{ $d->{$job}{hosts} };
-    my @nprocs = @{ $d->{$job}{nproc} };
-    my $nprocs = $nprocs[0];
+    my @hosts = @{ $d->{$job}{hosts} };
 
     config_set_internal( 'pbs_server', $d->{$job}{server} );
 
     my %pcmd;
 
-    $pcmd{nprocesses} = $nprocs;
+    $pcmd{nprocesses} = $d->{$job}{nproc};
     $pcmd{nhosts}     = @hosts;
     @{ $pcmd{host_list} } = @hosts;
 


-- 

Ashley Pittman, Bath, UK.

Padb - A parallel job inspection tool for cluster computing
http://padb.pittman.org.uk



-------------- next part --------------
An HTML attachment was scrubbed...
URL: 

From padb at googlecode.com  Thu Nov 26 10:31:02 2009
From: padb at googlecode.com (padb at googlecode.com)
Date: Thu, 26 Nov 2009 10:31:02 +0000
Subject: [padb] r341 committed - Store nprocs for pbs jobs in a variable
	rather than a single-length ar...
Message-ID: <000e0cd148be00635b047943ac2d@google.com>

Revision: 341
Author: apittman
Date: Thu Nov 26 02:30:48 2009
Log: Store nprocs for pbs jobs in a variable rather than a single-length  
array.

http://code.google.com/p/padb/source/detail?r=341

Modified:
  /trunk/src/padb

=======================================
--- /trunk/src/padb	Wed Nov 25 12:57:47 2009
+++ /trunk/src/padb	Thu Nov 26 02:30:48 2009
@@ -2662,8 +2662,7 @@
              my @champs = split(/\s+/);    # split by space
              if ( $champs[9] eq 'R' ) {    # take only Running
                  ($job) = split qr{\.}, $champs[0];
-                $nprocess = $champs[6];
-                push( @{ $pbs_tabjobs{$job}{nproc} }, $nprocess );
+                $pbs_tabjobs{$job}{nproc} = $champs[6];
              } else {
                  $job = undef;
              }
@@ -2732,15 +2731,13 @@
      my $job = shift;
      my $d   = pbs_get_data($target_user);

-    my @hosts  = @{ $d->{$job}{hosts} };
-    my @nprocs = @{ $d->{$job}{nproc} };
-    my $nprocs = $nprocs[0];
+    my @hosts = @{ $d->{$job}{hosts} };

      config_set_internal( 'pbs_server', $d->{$job}{server} );

      my %pcmd;

-    $pcmd{nprocesses} = $nprocs;
+    $pcmd{nprocesses} = $d->{$job}{nproc};
      $pcmd{nhosts}     = @hosts;
      @{ $pcmd{host_list} } = @hosts;




From thipadin.seng-long at bull.net  Mon Nov 30 14:36:00 2009
From: thipadin.seng-long at bull.net (thipadin.seng-long at bull.net)
Date: Mon, 30 Nov 2009 15:36:00 +0100
Subject: [padb] Patch of support of Slurm + Openmpi Orte manager
Message-ID: 

Hi Ashley,

May I introduce you my patch against padb r341 for supporting Slurm 
combined with openmpi orte manager.
The Key is we use salloc to get resource from slurm and then use it to run 
mpirun of openmpi to start jobs.
This kind of combination is not yet supported by current padb so far. When 
we start padb with
rmgr=slurm in a such job environment we have seen only the stack of orted 
(see below)
So my patch aims to remedy the situation.

Here are what's going on:
salloc -p tsl -w machu139,machu140,machu141
[thipa at machu0 padb_open]$ 
salloc: Granted job allocation 8324
[thipa at machu0 padb_open]$ 
[thipa at machu0 padb_open]$ srun -n1 mpirun -n 9 pp_sndrcv_spbl
srun: Warning: can't run 1 processes on 3 nodes, setting nnodes to 1
I am, process 0 starting on machu139, total by srun  9
I am, process 3 starting on machu139, total by srun  9
I am, process 6 starting on machu139, total by srun  9
I am, process 1 starting on machu140, total by srun  9
I am, process 8 starting on machu141, total by srun  9
I am, process 4 starting on machu140, total by srun  9
I am, process 2 starting on machu141, total by srun  9
I am, process 7 starting on machu140, total by srun  9
I am, process 5 starting on machu141, total by srun  9
Me, process 0, send  1000 to process 2

...........

[thipa at machu0 padb_open]$ squeue
  JOBID PARTITION     NAME     USER  ST       TIME  NODES NODELIST(REASON)
   8324       tsl     bash    thipa   R      36:33      3 machu[139-141]
[thipa at machu0 padb_open]$ 


padb with rmgr=slurm

[thipa at machu0 padb_open]$ ./padb -O rmgr="slurm" -O stack-shows-locals=no 
-O stack-shows-params=no --debug=verbose=all -tx 8324
DEBUG (verbose):   0: There are 1 processes over 3 hosts
-----------------
[0] (1 processes)
-----------------
main() at ?:?
  orterun() at ?:?
    opal_event_dispatch() at event.c:682
      opal_event_loop() at event.c:746
        poll_dispatch() at poll.c:167
          poll() at ?:?
DEBUG (verbose):   0: Completed command
[thipa at machu0 padb_open]$ 

padb with rmgr="sl-orte"  (my patch)


[thipa at machu0 padb_open]$ ./padb -O rmgr="sl-orte" -O 
stack-shows-locals=no  -O stack-shows-params=no --debug=verbose=all -tx 
8324
DEBUG (verbose):   0: There are 1 processes over 3 hosts
Warning, remote process state differs across ranks
state : ranks
R (running) : [2]
S (sleeping) : [0-1,3-8]
-----------------
[0-8] (9 processes)
-----------------
ThreadId: 1
  -----------------
  [0-1,3-8] (8 processes)
  -----------------
  main() at pp_sndrcv_spbl.c:55
    PMPI_Finalize() at pfinalize.c:46
      ompi_mpi_finalize() at runtime/ompi_mpi_finalize.c:224
        barrier() at grpcomm_bad_module.c:277
          opal_progress() at runtime/opal_progress.c:189
            ThreadId: 2
              start_thread() at ?:?
                btl_openib_async_thread() at btl_openib_async.c:346
                  poll() at ?:?
                    ThreadId: 3
                      start_thread() at ?:?
                        service_thread_start() at btl_openib_fd.c:427
                          select() at ?:?
  -----------------
  [2] (1 processes)
  -----------------
  main() at pp_sndrcv_spbl.c:50
    PMPI_Recv() at precv.c:78
      mca_pml_ob1_recv() at pml_ob1_irecv.c:104
        opal_progress() at runtime/opal_progress.c:207
          ThreadId: 2
            start_thread() at ?:?
              btl_openib_async_thread() at btl_openib_async.c:346
                poll() at ?:?
                  ThreadId: 3
                    start_thread() at ?:?
                      service_thread_start() at btl_openib_fd.c:427
                        select() at ?:?
DEBUG (verbose):   1: Completed command
[thipa at machu0 padb_open]$


Possibility to start jobs as follows:

1-salloc  ... mpirun  -n 6  openmpi_appli
2-salloc ....
    bash:  mpirun  -n 6  openmpi_appli
3-salloc ...
    bash:   srun -n 1 mpirun  -n 6  openmpi_appli

Here is the patch you may commit as is or work over. The patch support all 
possibility above.
I don't use scontrol listpids, because I found this command not a 
universal method (some version doesn't have it),
and may issued error message such as :
slurmd[machu139]: proctrack/pgid does not implement 
slurm_container_get_pids

Thipadin.
More later.





-------------- next part --------------
An HTML attachment was scrubbed...
URL: 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: slorte.patch
Type: application/octet-stream
Size: 4057 bytes
Desc: not available
URL: 

From ashley at pittman.co.uk  Mon Nov 30 16:31:51 2009
From: ashley at pittman.co.uk (Ashley Pittman)
Date: Mon, 30 Nov 2009 16:31:51 +0000
Subject: [padb] Patch of support of Slurm + Openmpi Orte manager
In-Reply-To: 
References: 
Message-ID: <1259598711.3532.24.camel@alpha>

On Mon, 2009-11-30 at 15:36 +0100, thipadin.seng-long at bull.net wrote:
> May I introduce you my patch against padb r341 for supporting Slurm
> combined with openmpi orte manager. 
> The Key is we use salloc to get resource from slurm and then use it to
> run mpirun of openmpi to start jobs. 

I knew you had to do this when running OpenMPI with slurm however I'd
never done it myself.  My test cluster has both installed so I should be
able to try it, do you happen to know if you need and special configure
options to either to allow this?

I'll try and get this running myself in the next couple of days but in
the mean time I've got some questions:

Does the mpirun job (i.e. the processes we want) have it's own slurm job
step or does it share the job step with the allocation?

I also notice the /proc/version in the patch, does this mean the patch
works on an OS other than Linux?

What happens if you run salloc... srun?  Does this work with the
existing support and how should users know which resource manager plugin
to pick (Ideally padb could do the right thing).

> [thipa at machu0 padb_open]$ ./padb -O rmgr="sl-orte" -O
> stack-shows-locals=no  -O stack-shows-params=no --debug=verbose=all
> -tx 8324 
> DEBUG (verbose):   0: There are 1 processes over 3 hosts 

This isn't great, the number of processes expected is so far only used
to check for missing processes but there are other potential uses for it
so I'd rather it was correct.

> I don't use scontrol listpids, because I found this command not a
> universal method (some version doesn't have it), 
> and may issued error message such as : 
> slurmd[machu139]: proctrack/pgid does not implement
> slurm_container_get_pids 

I'd prefer to use this if at all possible, this option was added at a
request my be several years ago so I'd have thought most versions have
it by now, can you be clearer on the versions where it doesn't work?

Ashley,
-- 

Ashley Pittman, Bath, UK.

Padb - A parallel job inspection tool for cluster computing
http://padb.pittman.org.uk




From ashley at pittman.co.uk  Mon Nov 30 17:47:32 2009
From: ashley at pittman.co.uk (Ashley Pittman)
Date: Mon, 30 Nov 2009 17:47:32 +0000
Subject: [padb] Better handling of threads in stack traces.
Message-ID: <1259603252.3532.111.camel@alpha>


I've been giving some thought to how to padb can handle threaded
applications better as the current scheme isn't ideal.

The current scheme works for single processes (ranks) and when used with
the -c or -C options to report output based on rank, where it falls
apart however is when stack traces are viewed in tree mode and this is
one of padb's strongest features.

There are several changes which would make sense I think, each having
their pros and cons so I'd like feedback on which people would prefer.

First would be to report extra threads in the same tree as the primary
thread, some magic would have to be applied to cover the fact that the
first thread in a process starts with main and subsequent ones start
with pthread_create() but this wouldn't be a insurmountable problem.
The big problem with this approach would be how to report thread
identifiers in the same rank-spec as rank rank identifiers, I could
revert to just using a list here but that doesn't work so well on big
systems.

The second option would be to treat each thread as a different entity
within the rank/process and have a number of trees displayed per job,
each dealing with a different thread, e.g. there would be a tree per
main thread and another tree for each extra thread encountered.  From a
technical perspective implementing this would require adding a namespace
to the {target_output} as it's passed back up the comms tree so is the
hardest to add but would probably lead to the best solution.

Finally there is the option of not showing all threads but allowing
users to select a single thread per invocation of padb.  This is the
simple but functional option although might be best viewed as a step
along the way to fully supporting multiple threads in future.  Here the
options are to be able to select threads by id (1,2,...) or perhaps by
having a white/black list of function names that should appear in the
stack for a thread before a thread is shown.

I'd welcome ideas on which people would prefer or if anybody has any
other thoughts on how to handle threads properly.

Ashley,

-- 

Ashley Pittman, Bath, UK.

Padb - A parallel job inspection tool for cluster computing
http://padb.pittman.org.uk







From ashley at pittman.co.uk  Mon Nov 30 19:10:57 2009
From: ashley at pittman.co.uk (Ashley Pittman)
Date: Mon, 30 Nov 2009 19:10:57 +0000
Subject: [padb] Release plans/roadmap.
Message-ID: <1259608257.3532.212.camel@alpha>


Apparently you can't have a proper software product without a published
road map so below are the release plans for the next release of padb.

The recent additions of allowing "mpirun" as a resource manager, Solaris
support and adding variables to tree based stack traces are to my mind
enough to justify a release on their own so I propose to wait no longer
than until the current batch of ideas are implemented.  Allowing a
couple of weeks for the code to settle this makes the best guess for the
date of a 3.1 feature release some time early in the new year, most
likely towards the end of February. 

I'd like to get the following issues addressed, be they bug fixes, new
features or general improvements to the package as a whole:

* Better threading support in stack traces (I've emailed about this
separately)

* Any new resource manager support from Bull

* Proper RPM/deb packaging

* Installation of minfo.x into $PREFIX/usr/lib/padb/

* Message queues being displayed by padb rather than minfo.  Padb will
still use minfo but rather than forward it's output directly padb will
parse the minfo output and then re-format this into something which
makes sense to the user.

* In-network reduce of collective state for deadlock detection.

* Adding a "Relay" option to allow padb to launch the inner processes
via a means other than the in-use resource manager.  none, pdsh, ssh and
orte are likely candidates for relay modes.

* Fixing some Quadrics bugs, mostly finding the install location of edb
and calculation of the shared memory key.

* Any other bug fixes that come along.

If anyone has other suggestions or problems they'd like to see addressed
before the next release I'd be keen to hear them.

Ashley,

-- 

Ashley Pittman, Bath, UK.

Padb - A parallel job inspection tool for cluster computing
http://padb.pittman.org.uk