From codesite-noreply at google.com Mon Jun 8 17:47:03 2009 From: codesite-noreply at google.com (codesite-noreply at google.com) Date: Mon, 08 Jun 2009 16:47:03 +0000 Subject: [padb-devel] [padb commit] r32 - Update the slurm link to the https version of their website. Message-ID: <001636ed657ae467d7046bd8fd91@google.com> Author: apittman Date: Mon Jun 8 09:46:18 2009 New Revision: 32 Modified: trunk/doc/index.html Log: Update the slurm link to the https version of their website. Modified: trunk/doc/index.html ============================================================================== --- trunk/doc/index.html (original) +++ trunk/doc/index.html Mon Jun 8 09:46:18 2009 @@ -50,7 +50,7 @@

What padb isn't

@@ -29,17 +29,17 @@ does not launch or wrap your jobs for you, it is not a job harness but rather attaches to or targets jobs which are already running. -

Licence

-padb is licenced under the +

License

+padb is licensed under the LGPL and as such is -open-source and free to use. +open-source and free to use and modify.

History

-Padb was origionally concieved by software developers at +Padb was originally conceived by software developers at Quadrics around 2004 to solve the kind of problems facing them at the time. It's been a part of the Quadrics software stack for a number of years and has recently been -made avalaible to a wider audience. It has been commercially supported +made available to a wider audience. It has been commercially supported for a number of years and is known to work at a scale of thousands of processes. @@ -53,10 +53,11 @@
  • Slurm
    • Runs natively on clusters running Slurm irrespective of the runtime used.
  • Open MPI
    • -Supports orte or OMPI jobs run under slurm. +Supports orte or OpenMPI jobs run under slurm.
  • MPICH2
    • Supports mpd or MPICH2 jobs run under slurm. +

    In addition padb can be told to target individual UNIX processes.

    Prerequisites

    Padb requires very little support from the OS or parallel Modified: trunk/doc/usage.html ============================================================================== --- trunk/doc/usage.html (original) +++ trunk/doc/usage.html Mon Jun 8 14:14:56 2009 @@ -2,55 +2,62 @@

    Selecting the Resource Manager (Job Launcher)

    Padb supports many resource managers and should select -the appropiate one for your cluster, if you have more than +the appropriate one for your cluster, if you have more than one resource manager installed or padb can't detect -the correct one use -O rmgr=<rmgs> +the correct one use the rmgr configuration option. -If no resource manager is found use -Ormgr=local and proccess -pids will be used instead of job ids. +

    If no resource manager is found use -O rmgr=local and process +identifiers will be used instead of job ids. + +

    The --list-rmgrs option can be used to show a list of detected +resource managers and their active jobs.

    Selecting the job(s) to target

    Padb provides several ways for the user to select which jobs -to target depending on requirments. All jobs detected by padb are -identified by a unique numberic "jobid", typically this maps directly +to target depending on requirements. All jobs detected by padb are +identified by a unique numeric "jobid", typically this maps directly only a identifier provided by the underlying resource manager. The default is to target jobs of the current user, this can be -over-ridden with the --user flag. +over-ridden with the --user flag.

    Showing list of current jobs

    To show a list of currently running jobs for a given user use the ---show-jobs flag. +--show-jobs flag.

    To target all jobs

    -To target all jobs currently running for a given user use the --all (-a) flag. +To target all jobs currently running for a given user use the --all (-a) flag.

    To target any jobs

    -To target "any" job currently running for a given user use the --any (-A) flag. -This differs from targeting all jobs as it will raise an error if more +To target "any" job currently running for a given user use the --any (-A) flag. +This differs from targeting all jobs as it will exit with an error if more than one job is running.

    To target a specific job

    To target a specific job specify the jobid for the job on the command line, after all other options. - -

    To target a specific job (Full report option)

    +

    To target a specific job (Full report option)

    If trying to diagnose a problem or gather information there is another -option, --full-report=<jobid>, this tells padb to target the job +option, --full-report=<jobid>, this tells padb to target the job specified and to report all information about the job it knows how to collect. This option is typically used when creating bug reports to -send to third parties or to inspect a job for anomolies. +send to third parties or to inspect a job for anomalies.

    Example

    To show a stack trace for process ID 10 use 
    padb -Ormgr=local --stack-trace 10
    +
    + +

    Selecting ranks (Processes)

    -In modes where data for each rank is reported seperatly it is possible -to restrict which ranks are queried, this is done via the --rank option. -Multiple ranks can be selected by specifing --rank multiple times. +In modes where data for each process is reported separately it is possible +to restrict which ranks are queried, this is done via the --rank option. +Multiple ranks can be selected by specifying --rank multiple times +or passing a comma separated list. +

    Selecting which mode to run in.

    Padb can present an array of different information about your @@ -60,42 +67,62 @@

    Full Report

    If you are just starting with padb or are creating an error -report for somebody else then the --full-report option is a good place +report for somebody else then the --full-report=<jobid> option is a good place to start, this will complete a full diagnostic report for the job iterating over the more common padb options. For large jobs this can generate a lot of output so redirecting to a file is recommended.

    Stack Trace

    -The --stack-trace (-x) option will cause a stack trace to be given for -each selected rank. The stack trace generated by this call is moderatly +The --stack-trace (-x) option will cause a stack trace to be given for +each selected rank. The stack trace generated by this call is moderately terse and gives one function call per line. This option is best combined -with --tree (-t) to condense stack traces into a tree based format. +with --tree (-t) to condense stack traces into a tree based format.

    Stack Trace (Longer)

    -Fuller stack traces can be obtained by setting the config options +Fuller stack traces can be obtained by setting the     configuration options stack-shows-params=1 and stack-shows-locals=1. This can help to narrow down a problem however does not work well with tree based stack traces -so is recommended to be used with the --rank flag. +so is recommended to be used with the --rank flag.

    Message Queues

    For MPI programs the MPI Message Queues can be shown on MPI stacks -which support it, use the --message-queue (-q) option to show the +which support it, use the --message-queue (-q) option to show the message queues. -

    Configuration options

    +

    Process info

    +To view information about individual processes in a job use the +--proc-info option, best combined with --proc=format=<format>, +for example --proc-format=vp,pid,state,pcpu,vmsize,vmrss + +
    + +

    Configuration options

    padb has a number of extended configuration options for tuning -it's behaviour, these can be set in a number of ways. All configuration +it's behavior, these can be set in a number of ways. All configuration options are key/value pairs. A list of configuration options along with their current values can be -obtained with the command "padb -Ohelp" +obtained by specifying a key of help with or without a value, i.e. padb -Ohelp + +

    Example

    +
    +ashley at alpha:~$ padb --show-jobs
+Error, multiple resource managers detected, use -Ormgr=
+orte slurm local-fd local
+ashley at alpha:~$ padb --show-jobs --config-option rmgr=orte
+59315
+ashley at alpha:~$ export PADB_RMGR=orte
+ashley at alpha:~$ padb --show-jobs
+59315
+ashley at alpha:~$
+
    From codesite-noreply at google.com Mon Jun 8 22:56:00 2009 From: codesite-noreply at google.com (codesite-noreply at google.com) Date: Mon, 08 Jun 2009 21:56:00 +0000 Subject: [padb-devel] [padb commit] r37 - Add a proc-shows-fds=full option to show the newly Message-ID: <000e0cd14eeec66f3a046bdd4eec@google.com> Author: apittman Date: Mon Jun 8 14:55:12 2009 New Revision: 37 Modified: trunk/src/padb Log: Add a proc-shows-fds=full option to show the newly exported flags and pos info for all file descriptors. Modified: trunk/src/padb ============================================================================== --- trunk/src/padb (original) +++ trunk/src/padb Mon Jun 8 14:55:12 2009 @@ -4774,11 +4774,40 @@ opendir( FDS, "$dir/fd" ); my @fds = readdir(FDS); closedir(FDS); + my @all_fddata; foreach my $fd (@fds) { next if ( $fd eq "." ); next if ( $fd eq ".." ); my $target = readlink "$dir/fd/$fd"; - output $vp, "fd$fd: $target"; + + my %fdhash; + $fdhash{target} = $target; + $fdhash{fd} = $fd; + + # New fdinfo data, it's verbose so only enable it + # if requested by -O proc-shows-fds=full + if ( $confInner{"proc-shows-fds"} eq "full" ) { + if ( -f "$dir/fdinfo/$fd" ) { + open( FDI, "$dir/fdinfo/$fd" ); + my @fdi = (); + close FDI; + foreach my $fdi (@fdi) { + chomp($fdi); + my ( $key, $value ) = split( ":", $fdi ); + $value =~ s/\t//g; + $fdhash{$key} = $value; + } + } + } + push( @all_fddata, \%fdhash ); + } + foreach my $fd (@all_fddata) { + if ( defined $fd->{pos} ) { + output( $vp, + "fd$fd->{fd}: $fd->{target} \($fd->{pos} $fd->{flags}\)" ); + } else { + output( $vp, "fd$fd->{fd}: $fd->{target}" ); + } } } if ( $confInner{"proc-shows-maps"} ) { From codesite-noreply at google.com Tue Jun 9 11:57:00 2009 From: codesite-noreply at google.com (codesite-noreply at google.com) Date: Tue, 09 Jun 2009 10:57:00 +0000 Subject: [padb-devel] [padb commit] r38 - Use symbolic names rather than numbers when delivering Message-ID: <000e0cd28d52e03a3e046be8372e@google.com> Author: apittman Date: Tue Jun 9 03:55:34 2009 New Revision: 38 Modified: trunk/src/padb Log: Use symbolic names rather than numbers when delivering a signal. Fixes issue #1. Modified: trunk/src/padb ============================================================================== --- trunk/src/padb (original) +++ trunk/src/padb Tue Jun 9 03:55:34 2009 @@ -201,6 +201,7 @@ use Sys::Hostname; use File::Temp qw(tempfile); use MIME::Base64; +use Config; ############################################################################### # @@ -566,6 +567,10 @@ my @config_options; my %ic_names; +# Populated in the outer args section so that outer code +# can access secondary comamnd line argunments by name. +my %secondary_args; + sub parse_args_outer { Getopt::Long::Configure("bundling"); @@ -627,6 +632,13 @@ $have_allfns_option++; } + # Put the args in a hash so that they can be referenced by name. + if ( defined $allfns{$mode}{secondary} ) { + foreach my $sec ( @{ $allfns{$mode}{secondary} } ) { + $secondary_args{ $sec->{arg_long} } = $sec->{value}; + } + } + return $mode; } @@ -3414,6 +3426,17 @@ ); } + # If delivering a signal check that it's valid. + if ( defined($mode) and ( $mode eq "kill" ) ) { + my $signal = $secondary_args{signal}; + my %sig_names; + map { $sig_names{$_} = 1 } split( " ", $Config{"sig_name"} ); + + if ( not defined $sig_names{$signal} ) { + cmdline_error("$prog: Error: signal $signal is invalid\n"); + } + } + if ( $tree and !( ( defined $mode && $mode eq "stack" ) or $input_file ) ) { cmdline_error("$prog: Error: --tree only works with --stack-trace\n"); } @@ -5652,6 +5675,7 @@ $confInner{"mode"} = $mode; + # Put the args in a hash so that they can be referenced by name. if ( defined $allfns{$mode}{secondary} ) { foreach my $sec ( @{ $allfns{$mode}{secondary} } ) { $confInner{"args"}{ $sec->{arg_long} } = $sec->{value}; @@ -5785,8 +5809,8 @@ 'secondary' => [ { 'arg_long' => 'signal', - 'type' => 'i', - 'default' => '15' + 'type' => 's', + 'default' => 'TERM' } ] }; From codesite-noreply at google.com Tue Jun 9 12:43:21 2009 From: codesite-noreply at google.com (codesite-noreply at google.com) Date: Tue, 09 Jun 2009 11:43:21 +0000 Subject: [padb-devel] [padb commit] r39 - Setup a find_pids callback for the inner to use when Message-ID: <000e0cd2e2a4a40652046be8dda1@google.com> Author: apittman Date: Tue Jun 9 04:42:53 2009 New Revision: 39 Modified: trunk/src/padb Log: Setup a find_pids callback for the inner to use when accessing the resource manager. Simplifies the code and should make it easier to add more resource managers in future. Modified: trunk/src/padb ============================================================================== --- trunk/src/padb (original) +++ trunk/src/padb Tue Jun 9 04:42:53 2009 @@ -33,7 +33,9 @@ # maintainer. # * Added "orte" to the list of resource managers supported # * Don't enable local-qsnet on non-qsnet systems. -# +# * inner_main() now uses callbacks for resource manager support. +# * --signal now takes names rather than numbers. + # Version 2.2 # * Add a --core-stack option along with --core and --exe to extract stack # traces from core files. @@ -165,12 +167,10 @@ # * Multi-pass argument handling, --kill also accepts --signal for example, # this should really be done at the getopt layer. Also proper usage # info for these secondary args. -# * inner_main() should possibly use callbacks for resource manager support, -# it's starting to look a bit messy. -# * find_slurm_pids() has some good code in it for finding parallel processes +# * slurm_find_pids() has some good code in it for finding parallel processes # this should be extrapolated out and so it can be used in the mpd case, # ideally on non-rms systems (RMS rocks in this regard) the rmgr callback -# should return a list of spawned pids and the code in find_slurm_pids() should +# should return a list of spawned pids and the code in slurm_find_pids() should # pass this tree to find the most interesting one. # * The mode {handler} functions should only be called once per node, it could then # correctly handle $confInner{gdb_file} and also attach to every process per node @@ -259,6 +259,7 @@ # job_to_key job key no Convert from jobId to shm key. # setup_pcmd job cmd|ncpus yes Command needed to launch shadow jobs. # cleanup_pcmd - - no Cleans up and temporary files. +# find_pids job - maybe Called on the inner to locate pids. # inner_rmgr var n/a no Resource manager to masquerade as. @@ -270,6 +271,7 @@ 'job_is_running' => \&rms_job_is_running, 'job_to_key' => \&rms_job_to_key, 'setup_pcmd' => \&rms_setup_pcmd, + 'find_pids' => \&rms_find_pids, }; $rmgr{"mpd"} = { @@ -277,6 +279,7 @@ 'get_active_jobs' => \&mpd_get_jobs, 'setup_pcmd' => \&mpd_setup_pcmd, 'cleanup_pcmd' => \&mpd_cleanup_pcmd, + 'find_pids' => \&mpd_find_pids, }; $rmgr{"orte"} = { @@ -284,6 +287,7 @@ 'get_active_jobs' => \&open_get_jobs, 'setup_pcmd' => \&open_setup_pcmd, 'cleanup_pcmd' => \&open_cleanup_pcmd, + 'find_pids' => \&open_find_pids, }; $rmgr{"lsf-rms"} = { @@ -298,12 +302,14 @@ 'get_active_jobs' => \&slurm_get_jobs, 'job_is_running' => \&slurm_job_is_running, 'setup_pcmd' => \&slurm_setup_pcmd, + 'find_pids' => \&slurm_find_pids, }; $rmgr{"local"} = { 'get_active_jobs' => \&local_get_jobs, 'job_is_running' => \&local_job_is_running, 'setup_pcmd' => \&local_setup_pcmd, + 'find_pids' => \&local_find_pids, }; $rmgr{"local-qsnet"} = { @@ -5414,7 +5420,7 @@ } # Do the right thing with slurm... -sub find_slurm_pids { +sub slurm_find_pids { my $jobid = shift; # Slurm has the concept of a "job" and a "job step" which are @@ -5509,7 +5515,7 @@ } # Local processes per node, i.e. no resource manager support. -sub find_local_pids { +sub local_find_pids { my $pid = shift; # Hard-wire this to vp 0, probably not true but without the resource manager it's difficult @@ -5522,7 +5528,7 @@ maybe_show_pid( $vp, $pid ); } -sub find_mpd_pids { +sub mpd_find_pids { my $job = shift; my $d = mpd_get_data(); @@ -5533,7 +5539,7 @@ } } -sub find_open_pids { +sub open_find_pids { my $job = shift; open_get_data( $confInner{"open-ps"} ); my $hostname = hostname(); @@ -5543,7 +5549,7 @@ } } -sub show_all_pids { +sub rms_find_pids { my $jobid = shift; my %vps; @@ -5703,6 +5709,10 @@ $confInner{"myld"} = $ENV{"LD_LIBRARY_PATH"}; + # $rjobid is used for accessing the stats on slurm + # systems, on rms it's just the jobId but on combined + # slurm/rms systems it's modifed to be the rms id + # and the jobid is left as the slurm job id. my $rjobid = $jobid; if ( exists $ENV{"SLURM_PROCID"} ) { $rjobid = get_rms_jobid($jobid); @@ -5719,25 +5729,23 @@ exit(0); } - if ( $confInner{"rmgr"} eq "local" ) { - - # Takes a pid. - find_local_pids($jobid); - - } elsif ( $confInner{"rmgr"} eq "mpd" ) { - find_mpd_pids($jobid); + # Handle resource managers better, simply call a callback + # as the outer does. + # As usual there is a special case, on Slurm systems + # running QsNet you can have the RMS kernel module loaded + # and these need to be handled differently so deal with + # them first and then go to the standard callback. - } elsif ( $confInner{"rmgr"} eq "orte" ) { - find_open_pids($jobid); - - } elsif ( -d "/proc/rms" ) { + if ( ( $confInner{rmgr} eq "slurm" ) and ( -d "/proc/rms" ) ) { # Takes a RMS job id. - show_all_pids($rjobid); + rms_find_pids($rjobid); } else { - - # Takes a native job id. - find_slurm_pids($jobid); + if ( not defined $rmgr{ $confInner{rmgr} }{find_pids} ) { + printf("Error, rmgr $confInner{rmgr} has no find_pids callback\n"); + exit(1); + } + $rmgr{ $confInner{rmgr} }{find_pids}($jobid); } if ( defined $allfns{$mode}{handler_all} ) { From codesite-noreply at google.com Tue Jun 9 14:08:48 2009 From: codesite-noreply at google.com (codesite-noreply at google.com) Date: Tue, 09 Jun 2009 13:08:48 +0000 Subject: [padb-devel] [padb commit] r40 - Make --full-report work better on non Quadrics systems by Message-ID: <001636e0a8262fbfe5046bea0f60@google.com> Author: apittman Date: Tue Jun 9 06:08:33 2009 New Revision: 40 Modified: trunk/src/padb Log: Make --full-report work better on non Quadrics systems by removing a lot of harmless error messages along the way. Modified: trunk/src/padb ============================================================================== --- trunk/src/padb (original) +++ trunk/src/padb Tue Jun 9 06:08:33 2009 @@ -35,6 +35,7 @@ # * Don't enable local-qsnet on non-qsnet systems. # * inner_main() now uses callbacks for resource manager support. # * --signal now takes names rather than numbers. +# * Check job is valid when using the --full-report option. # Version 2.2 # * Add a --core-stack option along with --core and --exe to extract stack @@ -458,8 +459,8 @@ XXXX --full-report=JOBID All of the above. - --nostrip-below-main Don't strip stack traces below main. - --nostrip-above-wait Don't strip stack traces about elan_waitWord. + --nostrip-below-main Don\'t strip stack traces below main. + --nostrip-above-wait Don\'t strip stack traces about elan_waitWord. --proc-format Specify information to show about processes. @@ -1430,7 +1431,7 @@ # vp's only, if it's not set then display a total for everyone. if ( not $d ) { - print("Statistics not valid\n"); + print("QsNet Statistics not valid\n"); return; } @@ -2366,6 +2367,11 @@ sub open_get_data { my ($filename) = @_; + # Simply return if called more than once. + if ( keys(%open_jobs) != 0 ) { + return; + + } my $hostname = hostname(); my $job; my @out; @@ -2379,6 +2385,11 @@ close OPEN; } + # Handle being called multiple times, zero the hash every + # time we are called. Of course we could just return the + # existing hash which might be quicker. + %open_jobs = (); + foreach my $l (@out) { chomp $l; next if ( $l eq "" ); @@ -2389,26 +2400,19 @@ } else { my @elems = split( /\|/, $l ); - # print "$#elems\X at elems\Y\n"; - if ( $#elems == 4 ) { + if ( $#elems == 6 ) { - #print "@elems\n"; - } elsif ( $#elems == 6 ) { - - #print "@elems\n"; my $host = $elems[4]; $host =~ s/ //g; $host =~ s/\t//g; next if $host eq "Node"; $open_jobs{$job}{hosts}{$host}++; - #print "Host is $host\n"; if ( $host eq $hostname ) { my $name = $elems[1]; $name =~ /\[\[(\d+)\,(\d+)\]\,(\d+)\]/; my $rank = $3; - # my $rank = $elems[2]; my $pid = $elems[3]; $rank =~ s/ //g; $pid =~ s/ //g; @@ -2417,14 +2421,11 @@ } } - # print "$_"; } if ( $conf{"verbose"} ) { print Dumper \%open_jobs; } - - # print keys %jobs; } sub open_get_jobs { @@ -2950,6 +2951,10 @@ my $errors = 0; + my $report_errors = 1; + + $report_errors = 0 if ($full_report); + my $pcmd = { pid => -1, in => "", @@ -2998,7 +3003,9 @@ my $handle = $pcmd->{err}; while (<$handle>) { my $line = $_; - print( STDERR "Error ($jobid,$mode): $line" ); + if ($report_errors) { + print( STDERR "Error ($jobid,$mode): $line" ); + } $errors++; } @@ -3015,7 +3022,10 @@ if ( $res != 0 ) { my %status = rc_status($res); if ( job_is_running($jobid) ) { - printf("Failed to run parallel command (rc = $status{rc})\n"); + if ($report_errors) { + printf( + "Failed to run parallel command (rc = $status{rc})\n"); + } } else { printf("Job $jobid is no longer active\n"); return 1; @@ -3333,6 +3343,14 @@ } if ($full_report) { + + if ( not job_is_running($full_report) ) { + printf( STDERR +"Job $full_report is not active, use --show-jobs to see active jobs\n" + ); + exit(1); + } + printf("padb version $version\n"); printf("full job report for job $full_report\n\n"); @@ -3342,7 +3360,7 @@ my $res; $stats_total = 1; $group = 1; - $res = go_job( $full_report, undef ); + $res = go_job( $full_report, "full-report" ); undef $stats_total; undef $group; From codesite-noreply at google.com Tue Jun 9 14:23:51 2009 From: codesite-noreply at google.com (codesite-noreply at google.com) Date: Tue, 09 Jun 2009 13:23:51 +0000 Subject: [padb-devel] [padb commit] r41 - Remove a very old function which is no longer called. Message-ID: <000e0cd179e20e15c4046bea4544@google.com> Author: apittman Date: Tue Jun 9 06:22:39 2009 New Revision: 41 Modified: trunk/src/padb Log: Remove a very old function which is no longer called. Modified: trunk/src/padb ============================================================================== --- trunk/src/padb (original) +++ trunk/src/padb Tue Jun 9 06:22:39 2009 @@ -2693,31 +2693,6 @@ } } -sub stack_to_hash { - my $line = shift; - if ( $line =~ /([\w\?]*)\(\) at ([\?\w\/\.]*):(\d*)/ ) { - my %data; - $data{func} = $1; - $data{file} = $2; - $data{line} = $3; - return %data; - } - return undef; -} - -sub munge_stack_traces { - my $lines = shift; - - foreach my $tag ( keys %$lines ) { - for ( my $l = 0 ; $l <= $#{ $lines->{$tag} } ; $l++ ) { - my %data = stack_to_hash( $lines->{$tag}->[$l] ); - if ( %data and $data{func} ) { - $lines->{$tag}->[$l] = "$data{func}()"; - } - } - } -} - sub sort_proc_hashes { my $key = shift; my @all = @_; @@ -2768,10 +2743,6 @@ if ( $strip_below_main or $strip_above_wait ) { strip_stack_traces($lines); } - - #if ( defined $conf{stack_format} ) { - # munge_stack_traces($lines); - #} } if ($tree) { From codesite-noreply at google.com Wed Jun 10 14:44:04 2009 From: codesite-noreply at google.com (codesite-noreply at google.com) Date: Wed, 10 Jun 2009 13:44:04 +0000 Subject: [padb-devel] [padb commit] r42 - Write the extensions page and add my email to the front Message-ID: <000e0cd2dda43037b3046bfeabcc@google.com> Author: apittman Date: Wed Jun 10 06:43:10 2009 New Revision: 42 Modified: trunk/doc/extensions.html trunk/doc/header.html trunk/doc/index.html Log: Write the extensions page and add my email to the front page. Modified: trunk/doc/extensions.html ============================================================================== --- trunk/doc/extensions.html (original) +++ trunk/doc/extensions.html Wed Jun 10 06:43:10 2009 @@ -1,10 +1,109 @@ -

    Patches

    -The advanced Group Deadlock Detection algorithm within padb -requires modifications to the MPI library to function properly. -At this time patches are available for Open MPI svn trunk -only. -Contact the developer mailing list -for more information. + +

    MPI collective debugger extension proposal

    + +

    Overview

    +The current +MPI debugger interface +is used to export information from a running application to a debugger. The current +interface allow the debugger to look at a MPI Process, to iterate over communicators +within that process and to view message queues associated with a communicator. + +

    +I propose an extension to this to export information about individual communicators +within a process, in particular information about collective operations (MPI_Bcast, +MPI_Reduce et. al.) + +

    Implementation

    +The specific information I propose is to add a communicator specific counter for +each possible collective where the counter simply records the number of times the +collective has been called on this communicator. Along with this is keeping a second piece +of data, that of if the process is still performing the collective operation. + +

    + +A new enum is added to the interface, mqs_comm_class with values for each collective +call. + +

    + +A single extra callback function mqs_get_comm_coll_state is added to the +interface and queries the current communicator in the same way as mqs_next_operation. +This function takes the standard process parameter, a mqs_comm_class enum as input +for which collective to query and two int *, the first of these is a pointer to a +int set which should be set to the count of the number of calls to the collective, the second +is a pointer to a int which should be set zero or one depending if the collective operation is still +active. + +

    + +A successful call to the mqs_get_comm_coll_state should return mqs_ok with +mqs_no_information being used in the case where information isn't available. This allows +further enum values to be added in the future should the mpi-forum approve new collective +functions without needing to change the debugger function interface. + +

    Performance Impact

    +Maintaining this data does add code to the "critical path" of the MPI stack, in +it's simplest form all it requires is a pair of counter increments per collective call, +one on function entry and one on function exit so whilst there is a non-zero run-time cost +associated with maintaining this information it's a minimal one. + +

    mpi_interface.h

    +The additions required to mpi_interface.h are shown below. +
    +
+typedef enum
+{
+  mqs_comm_barrier,
+  mqs_comm_broadcast,
+  mqs_comm_allgather,
+  mqs_comm_allgatherv,
+  mqs_comm_allreduce,
+  mqs_comm_alltoall,
+  mqs_comm_alltoallv,
+  mqs_comm_reduce_scatter,
+  mqs_comm_reduce,
+  mqs_comm_gather,
+  mqs_comm_gatherv,
+  mqs_comm_scan,
+  mqs_comm_scatter,
+  mqs_comm_scatterv
+} mqs_comm_class;
+
+/***********************************************************************
+ * Collective extension
+ *
+ * This extension should be considered optional and the debugger should
+ * correctly the case where it doesn't exist.
+ *
+ */
+
+/*
+ * Return the state of collective operations for the currently active
+ * communicator, that is the number of times the collective has been
+ * called and if the operation is still in progress.
+ *
+ * The first int is *really* mqs_comm_class.
+ */
+extern int mqs_get_comm_coll_state (mqs_process *, int, int *, int *);
+
    + +

    Benefits

    +The extension allows a debugger or external program to know the state of collective +calls with the parallel program. In the typical scenario of debugging a hung +application this knowledge allows the debugger and programmer to know instantly +which processes are stuck in collective calls and which aren't, either because they +have successful made the collective call and returned or because they haven't +made the calls other ranks in a communicator have. This information allows swift +identification of problem areas within the job where further investigation may be +required. + +

    +This extension was originally developed in early 2007 whilst I was working at +Quadrics and has proved it's value numerous times in real-life cases. + +

    Sample Implementation

    +At this time a sample implementation is available for OpenMPI only although work +is being done a MPICH2 version.

    Patch for OpenMPI trunk. Modified: trunk/doc/header.html ============================================================================== --- trunk/doc/header.html (original) +++ trunk/doc/header.html Wed Jun 10 06:43:10 2009 @@ -1,7 +1,7 @@ Padb: A parallel debugging tool

    -

    Parallel Application Discovery Browser

    +

    Parallel Application Debugger

    Padb  usage  download  Modified: trunk/doc/index.html ============================================================================== --- trunk/doc/index.html (original) +++ trunk/doc/index.html Wed Jun 10 06:43:10 2009 @@ -9,6 +9,9 @@ open source, non-interactive, command line, script-able tool intended for use by programmers and system administrators alike. +

    +Padb is currently maintained outside of Quadrics by Ashley Pittman +

    Features

    A list of configuration options along with their current values can be obtained by specifying a key of help with or without a value, i.e. padb -Ohelp

    Example

    + + +
      ashley at alpha:~$ padb --show-jobs
  Error, multiple resource managers detected, use -Ormgr=
@@ -126,3 +104,5 @@
  59315
  ashley at alpha:~$
    +
    From codesite-noreply at google.com Mon Jun 15 21:46:00 2009 From: codesite-noreply at google.com (codesite-noreply at google.com) Date: Mon, 15 Jun 2009 20:46:00 +0000 Subject: [padb-devel] [padb commit] r50 - Add documentation of the kill mode and make sure Message-ID: <001636163cc74dce95046c692513@google.com> Author: apittman Date: Mon Jun 15 13:38:38 2009 New Revision: 50 Modified: trunk/doc/modes.html trunk/doc/upload_website Log: Add documentation of the kill mode and make sure we upload the ompi patch to the correct directory. Modified: trunk/doc/modes.html ============================================================================== --- trunk/doc/modes.html (original) +++ trunk/doc/modes.html Mon Jun 15 13:38:38 2009 @@ -27,7 +27,7 @@

    Padb can be told to loop, performing the same query over and over again. This is enabled by the --watch flag -and further controlled by --config-option=interval=<seconds> and --config-option=watch-clears-screen=<bool> +and further controlled by --config-option interval=<seconds> and --config-option watch-clears-screen=<bool> options, the default values for these options are 10 and 1 respectively.

    @@ -178,9 +178,14 @@

    MPI Specific modes

    -

    MPI message queues

     -The option --message-queue will tell padb to read the MPI message +

    MPI message queues

     + +The option --mpi-queue will tell padb to read the MPI message queues from your application if possible. Here shown with the --compress option. + +

    +The option --message-queue shows the tport queues on QsNet systems. On non-QsNet systems it automatically falls back +to --mpi-queue. 
    @@ -272,6 +277,10 @@
    + +

    Signal delivery

    +To deliver signals to processes in a job use the --kill mode together with the optional +--signal=<name>> option. No output is produced by this mode.

    Quadrics specific modes

    The --set-debug, --group and --statistics modes are QsNet specific. The --deadlock mode Modified: trunk/doc/upload_website ============================================================================== --- trunk/doc/upload_website (original) +++ trunk/doc/upload_website Mon Jun 15 13:38:38 2009 @@ -22,7 +22,7 @@ done ftp-upload --host padb.pittman.org.uk -u padb at pittman.co.uk --password $PASSWORD OpenMPI-padb-groups.patch -ftp-upload --host padb.pittman.org.uk -u padb at pittman.co.uk --password $PASSWORD --as download/OpenMPI-padb-groups.patch OpenMPI-padb-groups.patch +ftp-upload --host padb.pittman.org.uk -u padb at pittman.co.uk --password $PASSWORD --as extensions/OpenMPI-padb-groups.patch OpenMPI-padb-groups.patch echo All done. exit 0 From codesite-noreply at google.com Tue Jun 16 11:28:42 2009 From: codesite-noreply at google.com (codesite-noreply at google.com) Date: Tue, 16 Jun 2009 10:28:42 +0000 Subject: [padb-devel] [padb commit] r51 - Fix a corner case where the stack-trace "tree" mode Message-ID: <0016e64135de820971046c74a331@google.com> Author: apittman Date: Tue Jun 16 03:12:06 2009 New Revision: 51 Modified: trunk/src/padb Log: Fix a corner case where the stack-trace "tree" mode was dropping frames. If one process in the job had a short stack further processes in the job erronously got reported as short too. Modified: trunk/src/padb ============================================================================== --- trunk/src/padb (original) +++ trunk/src/padb Tue Jun 16 03:12:06 2009 @@ -1784,7 +1784,7 @@ # hash containing the "txt" of the function name and a further array # of hash references called "children". sub go_p { - my ( $level, $indent, $indentNo, $lines, @tags ) = @_; + my ( $level, $lines, @tags ) = @_; my @peers; my $prev; @@ -1795,7 +1795,6 @@ return if ( !defined($tag) ); return if ( !defined( $lines->{$tag} ) ); - return if ( !defined( $lines->{$tag}[$level] ) ); my @identical = (); my @different = (); @@ -1817,7 +1816,12 @@ } } } else { - @different = @tags; + foreach my $dtag (@tags) { + if ( $dtag != $tag ) { + push( @different, $dtag ); + } + } + } # Move $endlevel on as far as we can... @@ -1846,10 +1850,10 @@ $endlevel = ( $#{ $lines->{$tag} } ); } - printf( "level $level, endlevel $endlevel, identical $#identical\n", ) - if $conf{"tree-verbose"}; + printf( +"level $level, endlevel $endlevel, identical:@identical different:@different\n", + ) if $conf{"tree-verbose"}; - my $i = $indent; for ( my $l = $level ; $l <= $endlevel ; $l++ ) { my %this; @@ -1873,8 +1877,7 @@ unshift @identical, $tag; } - $prev->{children} = - go_p( $endlevel + 1, "$i", $indentNo + 1, $lines, @identical ); + $prev->{children} = go_p( $endlevel + 1, $lines, @identical ); } printf( @@ -1882,11 +1885,12 @@ ) if $conf{"tree-verbose"}; if (@different) { - my $new = go_p( $level, "$indent", $indentNo, $lines, @different ); + my $new = go_p( $level, $lines, @different ); foreach my $n ( @{$new} ) { push @peers, $n; } } + return \@peers; } @@ -2886,7 +2890,7 @@ } if ($tree) { - print show_tree go_p( 0, "", 0, $lines, + print show_tree go_p( 0, $lines, ( sort { $a <=> $b } ( keys %$lines ) ) ); } elsif ($compress) { foreach my $tag ( sort { $a <=> $b } ( keys %$lines ) ) { From codesite-noreply at google.com Tue Jun 16 13:18:36 2009 From: codesite-noreply at google.com (codesite-noreply at google.com) Date: Tue, 16 Jun 2009 12:18:36 +0000 Subject: [padb-devel] [padb commit] r52 - Add the --proc-summary mode to the changelog. Message-ID: <0016368e1c2f94a9f2046c762cfb@google.com> Author: apittman Date: Tue Jun 16 03:13:44 2009 New Revision: 52 Modified: trunk/src/padb Log: Add the --proc-summary mode to the changelog. Modified: trunk/src/padb ============================================================================== --- trunk/src/padb (original) +++ trunk/src/padb Tue Jun 16 03:13:44 2009 @@ -36,6 +36,8 @@ # * inner_main() now uses callbacks for resource manager support. # * --signal now takes names rather than numbers. # * Check job is valid when using the --full-report option. +# * Add a --proc-summary option to replace --proc-info --proc-format +# This gives a very efficient "job top" program. # Version 2.2 # * Add a --core-stack option along with --core and --exe to extract stack From codesite-noreply at google.com Tue Jun 16 13:46:14 2009 From: codesite-noreply at google.com (codesite-noreply at google.com) Date: Tue, 16 Jun 2009 12:46:14 +0000 Subject: [padb-devel] [padb commit] r53 - Quick hack to make --deadlock work with input files. Message-ID: <0016e645ab305efcc6046c768f97@google.com> Author: apittman Date: Tue Jun 16 05:45:35 2009 New Revision: 53 Modified: trunk/src/padb Log: Quick hack to make --deadlock work with input files. The way modes and output formatting works is due to change soon so just make it work for now and tidy it up later. Modified: trunk/src/padb ============================================================================== --- trunk/src/padb (original) +++ trunk/src/padb Tue Jun 16 05:45:35 2009 @@ -3554,8 +3554,12 @@ $countoutput = 1 if ( ( defined $mode and $mode eq "stack" ) or $conf{"verbose"} ); - if ($input_file) { - go_file( $input_file, "input" ); + if ( defined($input_file) ) { + my $m = "input"; + if ( defined $mode ) { + $m = $mode; + } + go_file( $input_file, $m ); exit(0); } @@ -4417,7 +4421,17 @@ sub deadlock_detect { my ( $handle, $lines ) = @_; - my $data = $lines->{raw}; + my $data; + + # XXX This is a bit of a hack to make the deadlock + # code work with input files, the whole thing is due + # a tidy-up on the full-duplex branch where this should + # be solved properly. + if ( defined $lines->{raw} ) { + $data = $lines->{raw}; + } else { + $data = $lines->{lines}; + } # print Dumper $data; my %coll_data; From codesite-noreply at google.com Tue Jun 16 16:20:39 2009 From: codesite-noreply at google.com (codesite-noreply at google.com) Date: Tue, 16 Jun 2009 15:20:39 +0000 Subject: [padb-devel] [padb commit] r55 - Create a full-duplex branch Message-ID: <0016e645ab30999976046c78b765@google.com> Author: apittman Date: Tue Jun 16 07:56:43 2009 New Revision: 55 Added: branches/full-duplex/ - copied from r54, /trunk/ Log: Create a full-duplex branch From codesite-noreply at google.com Tue Jun 16 16:57:13 2009 From: codesite-noreply at google.com (codesite-noreply at google.com) Date: Tue, 16 Jun 2009 15:57:13 +0000 Subject: [padb-devel] [padb commit] r54 - Add a branches directory for storing branches on. Message-ID: <0016e645400665de73046c793a9b@google.com> Author: apittman Date: Tue Jun 16 07:55:26 2009 New Revision: 54 Added: branches/ Log: Add a branches directory for storing branches on. From codesite-noreply at google.com Wed Jun 17 12:01:38 2009 From: codesite-noreply at google.com (codesite-noreply at google.com) Date: Wed, 17 Jun 2009 11:01:38 +0000 Subject: [padb-devel] [padb commit] r57 - Enable warnings for the Branch and fix anything it throws up. Message-ID: <0016364ecdfc26abb7046c8937cd@google.com> Author: apittman Date: Wed Jun 17 03:26:07 2009 New Revision: 57 Modified: branches/full-duplex/src/padb Log: Enable warnings for the Branch and fix anything it throws up. Modified: branches/full-duplex/src/padb ============================================================================== --- branches/full-duplex/src/padb (original) +++ branches/full-duplex/src/padb Wed Jun 17 03:26:07 2009 @@ -1,4 +1,4 @@ -#!/usr/bin/perl +#!/usr/bin/perl -w # padb. a simple parallel debugging aid from Quadrics. @@ -2862,7 +2862,6 @@ } foreach my $hash (@all) { my @res; - my @res; foreach my $key (@proc_format_array) { my $value = "??"; if ( defined $hash->{$key} ) { @@ -5083,7 +5082,7 @@ chomp $l; my @stats = split( / /, $l ); for ( my $i = 0 ; $i <= $#stats ; $i++ ) { - output( $vp, "$prefix.$stat_names[$i]: @stats[$i]" ); + output( $vp, "$prefix.$stat_names[$i]: $stats[$i]" ); } } @@ -6105,7 +6104,7 @@ $res->{request} = $cmd; } - if ( $cmd->{quit} == 1 ) { + if ( defined( $cmd->{quit} ) and ( $cmd->{quit} == 1 ) ) { exit(0); } @@ -6188,8 +6187,7 @@ $netdata->{server} = $server; $netdata->{key} = $key; - my $sel = $netdata->{sel}; - my $server = $netdata->{server}; + my $sel = $netdata->{sel}; while ( my @data = $sel->can_read() ) { foreach my $s (@data) { From codesite-noreply at google.com Wed Jun 17 12:21:52 2009 From: codesite-noreply at google.com (codesite-noreply at google.com) Date: Wed, 17 Jun 2009 11:21:52 +0000 Subject: [padb-devel] [padb commit] r56 - First code to implement bi-directional communication Message-ID: <0016e644d5aa89d28b046c897f0a@google.com> Author: apittman Date: Wed Jun 17 02:47:07 2009 New Revision: 56 Modified: branches/full-duplex/src/padb Log: First code to implement bi-directional communication between the outer and the inner processes, currently it only works for one node and it only works for --mpi-watch. Modified: branches/full-duplex/src/padb ============================================================================== --- branches/full-duplex/src/padb (original) +++ branches/full-duplex/src/padb Wed Jun 17 02:47:07 2009 @@ -205,6 +205,8 @@ use File::Temp qw(tempfile); use MIME::Base64; use Config; +use IO::Socket; +use IO::Select; ############################################################################### # @@ -377,6 +379,7 @@ $conf{"local-fd-name"} = "/dev/null"; $conf{"stack-strip-above"} = "elan_waitWord,elan_pollWord,elan_deviceCheck,opal_condition_wait"; +$conf{"full-duplex"} = 0; # $conf{stack-format} = undef; @@ -2779,8 +2782,8 @@ # print Dumper $lines; my $s = ""; - foreach my $l ( sort { $a <=> $b } ( keys %{ $lines->{raw} } ) ) { - $s .= $lines->{raw}{$l}{state}; + foreach my $l ( sort { $a <=> $b } ( keys %{ $lines->{from_vpid} } ) ) { + $s .= $lines->{from_vpid}{$l}{state}; } print("$s\n"); } @@ -3016,6 +3019,228 @@ } } +sub connect_to_child { + my ( $host, $port, $word ) = @_; + + my $socket = IO::Socket::INET->new( + PeerAddr => $host, + PeerPort => $port, + Proto => 'tcp', + ) or die("Failed to connect to child"); + + print $socket "hello $word\n"; + + #printf("Connecting to $host $port $word\n"); + return $socket; +} + +sub my_encode { + return encode_base64( nfreeze(shift), "" ); +} + +sub my_decode { + return thaw( decode_base64(shift) ); +} + +# We have read data on a socket, process it and call +# any callback. +sub extract_line { + my ( $handle, $sd ) = @_; + + my $str = $sd->{str}; + + # Do this to allow telnet sessions to work. + $str =~ s/\r//g; + + # printf("Testing $str\n"); + + if ( $str =~ /^([^\n]+)\n(.*)$/ ) { + + # printf("Calling callback for \"$1\"\n"); + $sd->{line_cb}( $handle, $sd, $1 ); + $sd->{str} = $2; + } else { + printf("No match\n"); + } + + return; + +} + +sub issue_command_to_inner { + my ( $cdata, $cmd ) = @_; + my $str = my_encode($cmd); + $cdata->{socket}->print("$str\n"); +} + +sub command_from_inner { + my ( $handle, $cdata, $line ) = @_; + + if ( $line eq "Welcome" ) { + + # printf("Sending data to all childen\n"); + # Tell all hosts to go. + + my $req; + $req->{mode} = $handle->{mode}; + $req->{cmd} = $handle->{cmd}; + $req->{jobid} = $handle->{jobid}; + $req->{cinner} = \%cinner; + + # print Dumper $req; + issue_command_to_inner( $cdata, $req ); + + return; + } + + # A reply from inner. + my $d = my_decode($line); + + $allfns{ $handle->{mode} }{out_handler}( undef, $d ); + + my $req; + $req->{quit} = 1; + $handle->{shutdown_in_progress} = 1; + issue_command_to_inner( $cdata, $req ); + + print Dumper $d if ( $conf{"full-duplex"} eq "debug" ); + +} + +sub go_parallel { + my $jobid = shift; + my $cmd = shift; + my $ncpus = shift; + my $raw = shift; + my $stats = shift; + my $mode = shift; + my $h = shift; + + my $errors = 0; + + my $report_errors = 1; + + my $pcmd = { + pid => -1, + in => "", + out => *OUT, + err => *ERR, + }; + + $pcmd->{pid} = open3( $pcmd->{in}, *OUT, *ERR, $cmd ) + or die "Unable to open3() pcmd: $!\n"; + + close $pcmd->{in}; + + my $comm_data; + + $comm_data->{mode} = $mode; + $comm_data->{cmd} = $cmd; + $comm_data->{jobid} = $jobid; + $comm_data->{shutdown_in_progress} = 0; + + my $sel = IO::Select->new(); + $sel->add( $pcmd->{out} ); + $sel->add( $pcmd->{err} ); + + while ( my @live = $sel->can_read() ) { + foreach my $h (@live) { + if ( $h eq $pcmd->{out} ) { + my $line = $h->getline(); + if ( not defined $line ) { + if ( not $comm_data->{shutdown_in_progress} ) { + printf("Warning, EOF from ofd\n"); + } + $sel->remove($h); + $h->close(); + next; + } + my @words = split( " ", $line ); + if ( $#words == 3 and $words[0] eq "connect" ) { + + my $socket = + connect_to_child( $words[1], $words[2], $words[3] ); + my $cdata; + $cdata->{active} = 1; + $cdata->{str} = ""; + $cdata->{socket} = $socket; + $cdata->{line_cb} = \&command_from_inner; + $comm_data->{sockets}{$socket} = $cdata; + $sel->add($socket); + next; + } + if ( $words[0] eq "debug" ) { + my $count = $sel->count(); + print("There are $count sockets\n"); + next; + } + print("inner: $line"); + } elsif ( $h eq $pcmd->{err} ) { + my $line = $h->getline(); + + if ( not defined $line ) { + if ( not $comm_data->{shutdown_in_progress} ) { + printf("Warning, EOF from efd\n"); + } + $sel->remove($h); + $h->close(); + next; + } + printf("einner:$line"); + } elsif ( defined $comm_data->{sockets}{$h} ) { + my $cdata = $comm_data->{sockets}{$h}; + + my $data; + my $nb = sysread( $h, $data, 1024 ); + + #printf("read $data ($nb) from fd\n"); + + if ( not defined $data or $nb == 0 ) { + if ( not $comm_data->{shutdown_in_progress} ) { + printf("EOF from child socket ($nb)\n"); + } + $sel->remove($h); + $h->close(); + next; + } + + $cdata->{str} .= $data; + extract_line( $comm_data, $cdata ); + + } else { + printf("Responce from unknown fd $h\n"); + exit(1); + } + } + my $count = $sel->count(); + if ( $count == 1 ) { + printf("All sockets closed?\n"); + } + } + + waitpid( $pcmd->{pid}, 0 ); + my $res = $?; + + printf("result from parallel command was $res\n") + if ( $conf{"verbose"} ); + + if ( $res != 0 ) { + my %status = rc_status($res); + if ( job_is_running($jobid) ) { + if ($report_errors) { + printf("Failed to run parallel command (rc = $status{rc})\n"); + } + } else { + printf("Job $jobid is no longer active\n"); + return 1; + } + } + + cleanup_pcmd(); + + exit(0); +} + sub go_job_once { my $jobid = shift; my $cmd = shift; @@ -3226,7 +3451,13 @@ sleep( $conf{"interval"} ); } } - my $errors = go_job_once( $jobid, $cmd, $ncpus, $raw, $stats, $mode, $h ); + my $errors; + if ( $conf{"full-duplex"} ) { + $errors = go_parallel( $jobid, "$cmd --full-duplex", + $ncpus, $raw, $stats, $mode, $h ); + } else { + $errors = go_job_once( $jobid, $cmd, $ncpus, $raw, $stats, $mode, $h ); + } cleanup_pcmd(); return $errors; } @@ -5505,6 +5736,17 @@ return \%res; } +sub mpi_watch_all { + my ($list) = @_; + my %res; + foreach my $proc ( @{$list} ) { + my $vp = $proc->{vp}; + my $pid = $proc->{pid}; + $res{$vp} = mpi_watch( $vp, $pid ); + } + return \%res; +} + sub show_pid { my ( $vp, $pid ) = @_; @@ -5854,6 +6096,154 @@ ); } +sub command_from_parent { + my $cmd = shift; + my $res; + + # This is only for debugging. + if ( $confInner{verbose} ) { + $res->{request} = $cmd; + } + + if ( $cmd->{quit} == 1 ) { + exit(0); + } + + # Setup the environment. + foreach my $key ( keys( %{ $cmd->{cinner} } ) ) { + $confInner{$key} = $cmd->{cinner}{$key}; + } + + $confInner{mode} = $cmd->{mode}; + + # Find the pids and register them all. + $rmgr{ $confInner{rmgr} }{find_pids}( $cmd->{jobid} ); + + # Now do the work. + $res->{from_vpid} = + $allfns{ $cmd->{mode} }{handler_all}( $confInner{"all-pids"} ); + + return $res; +} + +sub command_from_outer { + my ( $netdata, $cdata, $line ) = @_; + + my $s = $cdata->{socket}; + if ( not $cdata->{trusted} ) { + if ( $line eq "hello $netdata->{key}" ) { + + #printf("Trusting connection from $cdata->{desc}\n"); + $cdata->{trusted} = 1; + $cdata->{str} = ""; + $s->printf("Welcome\n"); + $netdata->{parent} = $cdata; + } elsif ( $line eq "debug" ) { + my $r = Dumper($netdata); + $s->printf($r); + $s->flush(); + $netdata->{sel}->remove($s); + $s->close(); + $cdata->{dead} = 1; + print("debug\n"); + } else { + + #printf("Closing connection from $cdata->{desc} (Bad signon)\n"); + $netdata->{sel}->remove($s); + $s->close(); + $cdata->{dead} = 1; + } + return; + } + + $cdata->{last_cmd} = my_decode($line); + if ( $netdata->{parent} eq $cdata ) { + my $res = command_from_parent( my_decode($line) ); + my $reply = my_encode($res); + $cdata->{socket}->printf("$reply\n"); + } +} + +sub connect_to_outer { + + my $server = IO::Socket::INET->new( + + Reuse => 1, + Proto => 'tcp', + LocalPort => 37132, + Listen => 2, + ) or die("not the best start"); + + my $lport = $server->sockport(); + my $hostname = hostname(); + my $key = "boris"; + my $signon_text = "connect $hostname $lport $key\n"; + + # For now just print the signon code to stdout and let the outer pick it up. + print($signon_text); + + my $netdata; + $netdata->{sel} = IO::Select->new(); + $netdata->{sel}->add($server); + $netdata->{server} = $server; + $netdata->{key} = $key; + + my $sel = $netdata->{sel}; + my $server = $netdata->{server}; + + while ( my @data = $sel->can_read() ) { + foreach my $s (@data) { + if ( $s == $server ) { + my $new = $server->accept() or die("Failed accept"); + $sel->add($new); + my $peer = getpeername($new); + my ( $port, $addr ) = unpack_sockaddr_in($peer); + my $ip = inet_ntoa($addr); + my $hostname = gethostbyaddr( $addr, AF_INET ); + + # printf "New connection from $hostname ($ip) $port\n"; + my %sinfo; + $sinfo{hostname} = $hostname; + $sinfo{trusted} = 0; + $sinfo{port} = $port; + $sinfo{desc} = "$hostname:$port"; + $sinfo{socket} = $new; + $sinfo{line_cb} = \&command_from_outer; + $netdata->{connections}{$new} = \%sinfo; + + # $new->printf("Hello from padb\n"); + #$new->autoflush(); + next; + } + + my $sinfo = $netdata->{connections}{$s}; + my $d; + sysread( $s, $d, 1024 ); + + # Dead connection. + if ( not defined $d ) { + printf("null read from $sinfo->{desc}\n"); + if ( eof($s) ) { + $sel->remove($s); + $s->close(); + $sinfo->{trusted} = 0; + $sinfo->{dead} = 1; + my $count = $sel->count(); + printf("EOF from $sinfo->{desc} $count sockets left\n"); + } + next; + } + + $sinfo->{str} .= $d; + extract_line( $netdata, $sinfo ); + + } + } + my $count = $sel->count(); + printf("Thats not supposed to happen count=($count)\n"); + +} + sub inner_main { $confInner{"slurm-job-step"} = "0"; @@ -5872,6 +6262,7 @@ my @config_options; my $line_formatted; my $jobid; + my $full_duplex; my %optionhash = ( "config-option|O=s" => \@config_options, @@ -5879,7 +6270,8 @@ "line-formatted" => \$line_formatted, "rank=i" => \@ranks, "stats-full" => \$stats, - "verbose|v+" => \$confInner{"verbose"} + "verbose|v+" => \$confInner{"verbose"}, + "full-duplex" => \$full_duplex, ); my %config_hash; @@ -5902,6 +6294,12 @@ GetOptions(%optionhash) or die("could not parse options\n"); + # If this works then nothing below here is needed as all + # requests can be sent over the socket. + if ($full_duplex) { + connect_to_outer(); + } + my $mode; foreach my $arg ( keys %config_hash ) { @@ -6113,7 +6511,7 @@ }; $allfns{mpi_watch} = { - 'handler' => \&mpi_watch, + 'handler_all' => \&mpi_watch_all, 'arg_long' => 'mpi-watch', 'help' => "Trace MPI programs", 'pre_out_handler' => \&pre_mpi_watch, From ashley at pittman.co.uk Thu Jun 18 09:35:34 2009 From: ashley at pittman.co.uk (Ashley Pittman) Date: Thu, 18 Jun 2009 09:35:34 +0100 Subject: [padb-devel] 2.5 Release candidate available Message-ID: <1245314134.4226.2.camel@localhost.localdomain> All, A 2.5-rc1 candidate is available for download. Assuming no problems are reported with this I'll make it into an official release Monday or Tuesday next week. http://padb.googlecode.com/files/padb-2.5-rc1.tgz Yours, Ashley Pittman. -- Ashley Pittman, Bath, UK. Padb - A parallel job inspection tool for cluster computing http://padb.pittman.org.uk From codesite-noreply at google.com Thu Jun 18 09:37:09 2009 From: codesite-noreply at google.com (codesite-noreply at google.com) Date: Thu, 18 Jun 2009 08:37:09 +0000 Subject: [padb-devel] [padb commit] r58 - 2.5 release candidate, change the version number Message-ID: <0016e644d0a6443278046c9b5014@google.com> Author: apittman Date: Thu Jun 18 01:29:09 2009 New Revision: 58 Modified: trunk/doc/download.html trunk/doc/index.html trunk/src/padb Log: 2.5 release candidate, change the version number and update the website to reflect this. Modified: trunk/doc/download.html ============================================================================== --- trunk/doc/download.html (original) +++ trunk/doc/download.html Thu Jun 18 01:29:09 2009 @@ -1,15 +1,24 @@

    Source code download

    -

    Beta version

    +

    Stable release

    + +

    Release Candidate

    +A 2.5-rc1 release candidate it avaliable for +download. +All being well this will lead to a stable release on or around the +23rd June. I would encourage all users to download this release for testing. -The current Beta release (r31) is 2.5-beta and available for download -here or from the google +

    + +Downloads can also me made direct from google +here or from the google downloads section. -

    Stable release

    +

    -At present there are no "stable" releases made since the Quadrics 2.2 -version, a release is scheduled for early June 2009. +This will be the first "stable" release since the Quarics 2.2 version.

    Quadrics release

    Modified: trunk/doc/index.html ============================================================================== --- trunk/doc/index.html (original) +++ trunk/doc/index.html Thu Jun 18 01:29:09 2009 @@ -10,7 +10,14 @@ for use by programmers and system administrators alike.

    -Padb is currently maintained outside of Quadrics by Ashley Pittman +Padb is currently maintained outside of Quadrics by +Ashley Pittman, +I will be attending ISC +later on this month. + +

    News

    +28-06-09 A 2.5-rc1 release candidate is avaliable to download from the + downloads page.

    Features