wrap_parallel {spaMM}

Selecting interfaces for parallelisation

Description

spaMM implements three interfaces for parallelisation. Depending on their arguments, either serial computation (default), a socket cluster (parallelisation default), or a fork cluster (available in linux and alike operating systems) can be used by all interfaces.

dopar is called by default by its bootstrap procedures, and dofuture has been developed as an alternative, whose use is controlled by spaMM.options(wrap_parallel="dofuture") (versus the default, spaMM.options(wrap_parallel="dopar"). combinepar is the third and more recent interface; it is not a formally supported wrap_parallel option because its additional functionalities are of no use in spaMM's bootstrap procedures.

dopar is based on a patchwork of backends: for socket clusters, depending whether the doSNOW package is attached, foreach or pbapply is called (doSNOW allows more efficient load balancing than pbapply); for fork clusters, parallel::mclapply is used. This makes it impossible to ensure consistency of options accross computation environments, notably of enforcing the .combine control of foreach; and this makes it error-prone to ensure identical control of random number generators in all cases (although dopar and combinepar still aim to ensure the latter control).

By contrast, dofuture is based only on the future and future.apply packages, in principle allowing a single syntax to control of random number generator across the different cases, hence repeatable results across them. This does not make a difference for bootstrap computations in spaMM as the bootstrap samples are never simulated in parallel: only refitting the models is performed in parallel, and fit results do not depend on random numbers. Further, the future-based code for socket clusters appears significantly slower than the one used by dopar. For these reasons, the latter function is used by default by spaMM.

combinepar is a third and more recent approach designed to address the other issue: it always uses foreach so that the .combine control is consistently enforced. It uses future only when no alternative is available to produce a progress bar (namely, for socket clusters when doSNOW is not available).