MPIRUN (1)
Run MPI programs on LAM nodes.
SYNTAX
0
mpirun
[-fhvO]
[-c <#> | -np <#>]
[-D | -wd <dir>]
[-ger | -nger]
[-c2c | -lamd]
[-nsigs]
[-nw | -w]
[-nx]
[-pty]
[-s <node>]
[-t | -toff | -ton]
[-x VAR1[=VALUE1][,VAR2[=VALUE2],...]]
[<nodes>]
<program> [-- <args>]
mpirun
[-fhvO]
[-D | -wd <dir>]
[-ger | -nger]
[-lamd | -c2c]
[-nsigs]
[-nw | -w]
[-nx]
[-pty]
[-t | -toff | -ton]
[-x VAR1[=VALUE1][,VAR2[=VALUE2],...]]
<schema>
b
1
OPTIONS
There are two forms of the
mpirun
command -- one for programs (i.e., SPMD-style applications), and one
for application schemas (see appschema(5)). Both forms of
mpirun
use the following options by default:
-c2c
-nger
-w .
These may each be overriden by their counterpart options, described
below.
Additionally,
mpirun
will send the name of the directory where it was invoked on the local
node to each of the remote nodes, and attempt to change to that
directory. See the "Current Working Directory" section, below.
-c <#>
Synonym for
-np
(see below).
-c2c
Use "client to client" (c2c) mode for MPI communication in the user
program. This mode can significantly speed up some applications, as
messages will be passed directly from the source rank to the
destination rank; the LAM daemons will not be used as third-party
message passing agents. However, this disables monitoring and
debugging capabilities; see MPI(7). This option is mutually exclusive
with
-lamd .
-D
Use the executable program location as the current working directory
for created processes. The current working directory of the created
processes will be set before the user's program is invoked. This
option is mutually exclusive with
-wd .
-f
Do not configure standard I/O file descriptors - use defaults.
-h
Print useful information on this command.
-ger
Enable GER (Guaranteed Envelope Resources) communication protocol
and error reporting. See MPI(7) for a description of GER. This
option is mutually exclusive with
-nger .
-lamd
Use the LAM "daemon mode" for MPI communication. See
-c2c
(above) and MPI(7) for a description of the "daemon mode"
communication.
-nger
Disable GER (Guaranteed Envelope Resources). This option is mutually
exclusive with
-ger .
-nsigs
Do not have LAM catch signals.
-np <#>
Run this many copies of the program on the given nodes. This option
indicates that the specified file is an executable program and not an
application schema. If no nodes are specified, all LAM nodes are
considered for scheduling; LAM will schedule the programs in a
round-robin fashion, "wrapping around" (and scheduling multiple copies
on a single node) if necessary.
-nw
Do not wait for all processes to complete before exiting
mpirun .
This option is mutually exclusive with
-w .
-nx
Do not automatically export LAM_MPI_* environment variables to the
remote nodes.
-O
Multicomputer is homogeneous. Do no data conversion when passing
messages.
-pty
Enable pseudo-tty support. Among other things, this enabled
line-buffered output (which is probably what you want). The only
reason that this feature is not enabled by default is because it is so
new and has not been extensively tested yet.
-s <node>
Load the program from this node. This option is not valid on the
command line if an application schema is specified.
-t, -ton
Enable execution trace generation for all processes. Trace generation
will proceed with no further action. These options are mutually
exclusive with
-toff .
-toff
Enable execution trace generation for all processes. Trace generation
will begin after processes collectively call MPIL_Trace_on(2). This
option is mutually exclusive with
-t
and
-ton .
-v
Be verbose; report on important steps as they are done.
-w
Wait for all applications to exit before
mpirun
exits.
-wd <dir>
Change to the directory <dir> before the user's program executes.
Note that if the
-wd
option appears both on the command line and in an application schema,
the schema will take precendence over the command line. This option
is mutually exclusive with
-D .
-x
Export the specified environment variables to the remote nodes before
executing the program. Existing environment variables can be
specified (see the Examples section, below), or new variable names
specified with corresponding values. The parser for the
-x
option is not very sophisticated; it does not even understand quoted
values. Users are advised to set variables in the environment, and
then use
-x
to export (not define) them.
-- <args>
Pass these runtime arguments to every new process. This must always
be the last argument to
mpirun .
This option is not valid on the command line if an application schema
is specified.
DESCRIPTION
One invocation of
mpirun
starts an MPI application running under LAM.
If the application is simply SPMD, the application can be specified on the
mpirun
command line.
If the application is MIMD, comprising multiple programs, an application
schema is required in a separate file.
See appschema(5) for a description of the application schema syntax,
but it essentially contains multiple
mpirun
command lines, less the command name itself. The ability to specify
different options for different instantiations of a program is another
reason to use an application schema.
To distinguish the two different forms,
mpirun
looks on the command line for <nodes> or the -c option. If
neither is specified, then the file named on the command line is
assumed to be an application schema. If either one or both are
specified, then the file is assumed to be an executable program. If
<nodes> and -c both are specified, then copies of the program
are started on the specified nodes according to an internal LAM
scheduling policy. Specifying just one node effectively forces LAM to
run all copies of the program in one place. If -c is given, but
not <nodes>, then all LAM nodes are used. If <nodes> is given, but
not -c, then one copy of the program is run on each node.
By default, LAM searches for executable programs on the target node
where a particular instantiation will run. If the file system is not
shared, the target nodes are homogeneous, and the program is
frequently recompiled, it can be convenient to have LAM transfer the
program from a source node (usually the local node) to each target
node. The -s option specifies this behavior and identifies the
single source node.
LAM looks for an executable program by searching the directories in
the user's PATH environment variable as defined on the source node(s).
This behavior is consistent with logging into the source node and
executing the program from the shell. On remote nodes, the "." path
is the home directory.
LAM looks for an application schema in three directories: the local
directory, the value of the LAMAPPLDIR environment variable, and
LAMHOME/boot, where LAMHOME is the LAM installation directory.
LAM directs UNIX standard input to /dev/null on all remote nodes. On
the local node that invoked
mpirun ,
standard input is inherited from
mpirun .
The default is what used to be the -w option to prevent conflicting
access to the terminal.
LAM directs UNIX standard output and error to the LAM daemon on all
remote nodes. LAM ships all captured output/error to the node that
invoked
mpirun
and prints it on the standard output/error of
mpirun .
Local processes inherit the standard output/error of
mpirun
and transfer to it directly.
Thus it is possible to redirect standard I/O for LAM applications by
using the typical shell redirection procedure on
mpirun .
% mpirun N my_app < my_input > my_output
The
-f
option avoids all the setup required to support standard I/O described
above. Remote processes are completely directed to /dev/null and
local processes inherit file descriptors from lamboot(1).
The
-pty
option enabled pseudo-tty support for process output. This allows,
among other things, for line buffered output from remote nodes (which
is probably what you want).
This option is not currently the default for
mpirun
because it has not been thoroughly tested on a variety of different
Unixes. Users are encouraged to use
-pty
and report any problems back to the LAM Team.
The default behavior of mpirun has changed with respect to the
directory that processes will be started in.
The
-wd
option to mpirun allows the user to change to an arbitrary directory
before their program is invoked. It can also be used in application
schema files to specify working directories on specific nodes and/or
for specific applications.
If the
-wd
option appears both in a schema file and on the command line, the
schema file directory will override the command line value.
The
-D
option will change the current working directory to the directory
where the executable resides. It cannot be used in application schema
files.
-wd
is mutually exclusive with
-D .
If neither
-wd
nor
-D
are specified, the local node will send the directory name where
mpirun was invoked from to each of the remote nodes. The remote nodes
will then try to change to that directory. If they fail (e.g., if the
directory does not exists on that node), they will start with from the
user's home directory.
All directory changing occurs before the user's program is invoked; it
does not wait until
MPI_INIT
is called.
Processes in the MPI application inherit their environment from the
LAM daemon upon the node on which they are running. The environment
of a LAM daemon is fixed upon booting of the LAM with lamboot(1) and
is inherited from the user's shell. On the origin node this will be
the shell from which lamboot(1) was invoked and on remote nodes this
will be the shell started by rsh(1). When running dynamically linked
applications which require the LD_LIBRARY_PATH environment variable to
be set, care must be taken to ensure that it is correctly set when
booting the LAM.
All environment variables that are named in the form LAM_MPI_* will
automatically be exported to new processes on the local and remote
nodes. This exporting may be inhibited with the
-nx
option.
Additionally, the
-x
option to
mpirun
can be used to export specific environment variables to the new
processes. While the syntax of the
-x
option allows the definition of new variables, note that the parser
for this option is currently not very sophisticated - it does not even
understand quoted values. Users are advised to set variables in the
environment and use
-x
to export them; not to define them.
Two switches control trace generation from processes running under LAM
and both must be in the on position for traces to actually be
generated. The first switch is controlled by
mpirun
and the second switch is initially set by
mpirun
but can be toggled at runtime with MPIL_Trace_on(2) and
MPIL_Trace_off(2). The -t (-ton is equivalent) and
-toff options all turn on the first switch. Otherwise the first
switch is off and calls to MPIL_Trace_on(2) in the application program
are ineffective. The -t option also turns on the second switch.
The -toff option turns off the second switch. See
MPIL_Trace_on(2) and lamtrace(1) for more details.
LAM's MPI library converts MPI messages from local representation to
LAM representation upon sending them and then back to local
representation upon receiving them. If the case of a LAM consisting
of a homogeneous network of machines where the local representation
differs from the LAM representation this can result in unnecessary
conversions. The -O switch can be used to indicate that the LAM
is homogeneous and turn off data conversion.
For much improved performance but much decreased observability, the
-c2c option directs LAM's MPI library to use the most direct
underlying mechanism to communicate with other processes, rather than
use the network message-passing of the LAM daemon. Unreceived
messages will be buffered in the destination process instead of the
LAM daemon. MPI process and message monitoring commands and tools
will be much less effective, usually reporting running processes and
empty message queues. Signal delivery with doom(1) is unaffected.
By default, LAM will guarantee a minimum amount of message envelope
buffering to each MPI process pair and will impede or report an error
to a process that attempts to overflow this system resource. This
robustness and debugging feature is implemented in a machine specific
manner when direct communication (-c2c) is used. For normal LAM
communication via the LAM daemon, a protocol is used. The -nger
option disables GER and the measures taken to support it. The minimum
GER is configured by the system administrator when LAM is installed.
See MPI(7) for more details.
EXAMPLES
mpirun N prog1
Load and execute prog1 on all nodes.
Search for the executable file on each node.
mpirun -c 8 prog1
Run 8 copies of prog1 wherever LAM wants to run them.
mpirun n8-10 -v -nw -s n3 prog1 -- -q
Load and execute prog1 on nodes 8, 9, and 10.
Search for prog1 on node 3 and transfer it to the three target nodes.
Report as each process is created.
Give "-q" as a command line to each new process.
Do not wait for the processes to complete before exiting
mpirun .
mpirun -v myapp
Parse the application schema, myapp, and start all processes specified
in it. Report as each process is created.
mpirun N N -pty -wd /workstuff/output -x DISPLAY run_app.csh
Run the application "run_app.csh" (assumedly a C shell script) twice
on each node in the system (ideal for 2-way SMPs). Also enable
pseudo-tty support, change directory to /workstuff/output, and export
the DISPLAY variable to the new processes (perhaps the shell script
will invoke an X application such as xv to display output).
mpirun -np 5 -D `pwd`/my_application
A common usage of
mpirun
in environments where a filesystem is shared between all nodes in the
multicomputer, using the shell-escaped "pwd" command specifies the
full name of the executable to run. This prevents the need for
putting the directory in the path; the remote notes will have an
absolute filename to execute (and change directory to it upon
invocation).
DIAGNOSTICS
mpirun: Exec format error
A non-ASCII character was detected in the application schema.
This is usually a command line usage error where
mpirun
is expecting an application schema and an executable file was given.
mpirun: syntax error in application schema, line XXX
The application schema cannot be parsed because of a usage or syntax error
on the given line in the file.
<filename>: No such file or directory
This error can occur in two cases.
Either the named file cannot be located or it has been found but
the user does not have sufficient permissions to execute the program
or read the application schema.
SEE ALSO
|
|