R: Reduce Operation for Batch Systems

batchReduce {batchtools}

R Documentation

Reduce Operation for Batch Systems

Description

A parallel and asynchronous Reduce for batch systems. Note that this function only defines the computational jobs. Each job reduces a certain number of elements on one slave. The actual computation is started with submitJobs. Results and partial results can be collected with reduceResultsList, reduceResults or loadResult.

Usage

batchReduce(
  fun,
  xs,
  init = NULL,
  chunks = seq_along(xs),
  more.args = list(),
  reg = getDefaultRegistry()
)

Arguments

`fun`	[`function(aggr, x, ...)`] Function to reduce `xs` with.
`xs`	[`vector`] Vector to reduce.
`init`	[ANY] Initial object for reducing. See `Reduce`.
`chunks`	[`integer(length(xs))`] Group for each element of `xs`. Can be generated with `chunk`.
`more.args`	[`list`] A list of additional arguments passed to `fun`.
`reg`	[`Registry`] Registry. If not explicitly passed, uses the default registry (see `setDefaultRegistry`).

Value

[data.table] with ids of added jobs stored in column “job.id”.

Examples


# define function to reduce on slave, we want to sum a vector
tmp = makeRegistry(file.dir = NA, make.default = FALSE)
xs = 1:100
f = function(aggr, x) aggr + x

# sum 20 numbers on each slave process, i.e. 5 jobs
chunks = chunk(xs, chunk.size = 5)
batchReduce(fun = f, 1:100, init = 0, chunks = chunks, reg = tmp)
submitJobs(reg = tmp)
waitForJobs(reg = tmp)

# now reduce one final time on master
reduceResults(fun = function(aggr, job, res) f(aggr, res), reg = tmp)