startMPIcluster {doMPI}

Create and start an MPI cluster

Description

The startMPIcluster function is used to start an MPI cluster.

Usage

startMPIcluster(count, verbose=FALSE, workdir=getwd(), logdir=workdir,
                maxcores=1, includemaster=TRUE, bcast=TRUE,
                comm=if (mpi.comm.size(0) > 1) 0 else 3,
                intercomm=comm + 1, mtag=10, wtag=11,
                defaultopts=list())

Arguments

count

Number of workers to spawn. If you start your script using mpirun, then you don't really need to use the count argument, because startMPIcluster will try to do something reasonable. To be more specific, if comm is 0, then it will set count to mpi.comm.size(0) - 1. In fact, it is an error to set count to any other value. If comm is greater than 0, it will determine the number of processes to spawn by calling mpi.universe.size(). If that value is greater than one, then count is set to one less. If that value is equal to one, then count is arbitrarily set to 2. Note that if you've started the script without mpirun, than mpi.universe.size() will always return 1, so count will default to 2.

verbose

Indicates if verbose messages should be enabled. Defaults to FALSE.

workdir

Working directory of the cluster workers. Defaults to the master's working directory.

logdir

Directory to put the worker log files. Defaults to workdir.

maxcores

Maximum number of cores for workers to use. Defaults to 1.

includemaster

Indicates if the master process should be counted as a load on the CPU. This will effect how many cores will be used on the local machine by mclapply, if a worker process is started on the local machine. Defaults to TRUE.

bcast

Indicates if a true MPI broadcast should be used to send shared “job” data to the workers. If FALSE is specified, the data is sent by separate messages to each worker, which is sometimes faster than using a broadcast. So this option really controls whether to do a real or an emulated broadcast. Defaults to TRUE.

comm

Communicator number to use. A value of 0 means to use non-spawn mode, which means the cluster workers are started using mpirun/ortrun with more than one worker. A value of 1 or more forces spawn mode. Multiple clusters can be started by using different values for comm and intercomm. It defaults to 0 if mpi.comm.size(0) > 1, otherwise 3.

intercomm

Inter-communicator number. Defaults to comm + 1.

mtag

Tag to use for messages sent to the master. Do not use this option unless you know what you're doing, or your program will very likely hang. Defaults to 10.

wtag

Tag to use for messages sent to the workers. Do not use this option unless you know what you're doing, or your program will very likely hang. Defaults to 11.

defaultopts

A list containing default values to use for some of the .options.mpi options. These options include: chunkSize, info, profile, bcastThreshold, forcePiggyback, nocompile, and seed.

Note

The startMPIcluster function will return an MPI cluster object of different classes, depending on the bcast option. This is because broadcasting is implemented as a method on the MPI cluster object, and that method is implemented differently in the different classes.

Also note that the bcast option has no effect if the backend-specific forcePiggyback option is used with foreach, since “piggy-backing” is an alternative way to send the job data to the workers in separate messages.

So there are currently three ways that the job data can be sent to the workers: piggy-backed with the first task to each worker, broadcast, or sent in separate messages. Which method is best will presumably depend on your hardware and your MPI implementation.

Examples

## Not run: 
# start and register an MPI cluster with two workers in verbose mode:
cl <- startMPIcluster(count=2, verbose=TRUE)
registerDoMPI(cl)
# and shut it down
closeCluster(cl)

# set the working directory to /tmp:
cl <- startMPIcluster(count=2, workdir='/tmp')
registerDoMPI(cl)
# and shut it down
closeCluster(cl)

## End(Not run)