R: Create a controller with an AWS Batch launcher.

crew_controller_aws_batch {crew.aws.batch}

R Documentation

Create a controller with an AWS Batch launcher.

Description

Create an R6 object to submit tasks and launch workers on AWS Batch workers.

Usage

crew_controller_aws_batch(
  name = NULL,
  workers = 1L,
  host = NULL,
  port = NULL,
  tls = crew::crew_tls(mode = "automatic"),
  tls_enable = NULL,
  tls_config = NULL,
  seconds_interval = 0.5,
  seconds_timeout = 60,
  seconds_launch = 1800,
  seconds_idle = Inf,
  seconds_wall = Inf,
  retry_tasks = TRUE,
  tasks_max = Inf,
  tasks_timers = 0L,
  reset_globals = TRUE,
  reset_packages = FALSE,
  reset_options = FALSE,
  garbage_collection = FALSE,
  launch_max = 5L,
  processes = NULL,
  aws_batch_config = list(),
  aws_batch_credentials = list(),
  aws_batch_endpoint = NULL,
  aws_batch_region = NULL,
  aws_batch_job_definition,
  aws_batch_job_queue,
  aws_batch_share_identifier = NULL,
  aws_batch_scheduling_priority_override = NULL,
  aws_batch_parameters = NULL,
  aws_batch_container_overrides = NULL,
  aws_batch_node_overrides = NULL,
  aws_batch_retry_strategy = NULL,
  aws_batch_propagate_tags = NULL,
  aws_batch_timeout = NULL,
  aws_batch_tags = NULL,
  aws_batch_eks_properties_override = NULL
)

Arguments

`name`	Name of the client object. If `NULL`, a name is automatically generated.
`workers`	Integer, maximum number of parallel workers to run.
`host`	IP address of the `mirai` client to send and receive tasks. If `NULL`, the host defaults to the local IP address.
`port`	TCP port to listen for the workers. If `NULL`, then an available ephemeral port is automatically chosen.
`tls`	A TLS configuration object from `crew_tls()`.
`tls_enable`	Deprecated on 2023-09-15 in version 0.4.1. Use argument `tls` instead.
`tls_config`	Deprecated on 2023-09-15 in version 0.4.1. Use argument `tls` instead.
`seconds_interval`	Number of seconds between polling intervals waiting for certain internal synchronous operations to complete, such as checking `mirai::status()`
`seconds_timeout`	Number of seconds until timing out while waiting for certain synchronous operations to complete, such as checking `mirai::status()`.
`seconds_launch`	Seconds of startup time to allow. A worker is unconditionally assumed to be alive from the moment of its launch until `seconds_launch` seconds later. After `seconds_launch` seconds, the worker is only considered alive if it is actively connected to its assign websocket.
`seconds_idle`	Maximum number of seconds that a worker can idle since the completion of the last task. If exceeded, the worker exits. But the timer does not launch until `tasks_timers` tasks have completed. See the `idletime` argument of `mirai::daemon()`. `crew` does not excel with perfectly transient workers because it does not micromanage the assignment of tasks to workers, so please allow enough idle time for a new worker to be delegated a new task.
`seconds_wall`	Soft wall time in seconds. The timer does not launch until `tasks_timers` tasks have completed. See the `walltime` argument of `mirai::daemon()`.
`retry_tasks`	`TRUE` to automatically retry a task in the event of an unexpected worker exit. `FALSE` to give up on the first exit and return a `mirai` error code (code number 19). `TRUE` (default) is recommended in most situations. Use `FALSE` for debugging purposes, e.g. to confirm that a task is causing a worker to run out of memory or crash in some other way.
`tasks_max`	Maximum number of tasks that a worker will do before exiting. See the `maxtasks` argument of `mirai::daemon()`. `crew` does not excel with perfectly transient workers because it does not micromanage the assignment of tasks to workers, it is recommended to set `tasks_max` to a value greater than 1.
`tasks_timers`	Number of tasks to do before activating the timers for `seconds_idle` and `seconds_wall`. See the `timerstart` argument of `mirai::daemon()`.
`reset_globals`	`TRUE` to reset global environment variables between tasks, `FALSE` to leave them alone.
`reset_packages`	`TRUE` to unload any packages loaded during a task (runs between each task), `FALSE` to leave packages alone.
`reset_options`	`TRUE` to reset global options to their original state between each task, `FALSE` otherwise. It is recommended to only set `reset_options = TRUE` if `reset_packages` is also `TRUE` because packages sometimes rely on options they set at loading time.
`garbage_collection`	`TRUE` to run garbage collection between tasks, `FALSE` to skip.
`launch_max`	Positive integer of length 1, maximum allowed consecutive launch attempts which do not complete any tasks. Enforced on a worker-by-worker basis. The futile launch count resets to back 0 for each worker that completes a task. It is recommended to set `launch_max` above 0 because sometimes workers are unproductive under perfectly ordinary circumstances. But `launch_max` should still be small enough to detect errors in the underlying platform.
`processes`	`NULL` or positive integer of length 1, number of local processes to launch to allow worker launches to happen asynchronously. If `NULL`, then no local processes are launched. If 1 or greater, then the launcher starts the processes on `start()` and ends them on `terminate()`. Plugins that may use these processes should run asynchronous calls using `launcher$async$eval()` and expect a `mirai` task object as the return value.
`aws_batch_config`	Named list, `config` argument of `paws.compute::batch()` with optional configuration details.
`aws_batch_credentials`	Named list. `credentials` argument of `paws.compute::batch()` with optional credentials (if not already provided through environment variables such as `AWS_ACCESS_KEY_ID`).
`aws_batch_endpoint`	Character of length 1. `endpoint` argument of `paws.compute::batch()` with the endpoint to send HTTP requests.
`aws_batch_region`	Character of length 1. `region` argument of `paws.compute::batch()` with an AWS region string such as `"us-east-2"`.
`aws_batch_job_definition`	Character of length 1, name of the AWS Batch job definition to use. There is no default for this argument, and a job definition must be created prior to running the controller. Please see https://docs.aws.amazon.com/batch/ for details. To create a job definition, you will need to create a Docker-compatible image which can run R and `crew`. You may which to inherit from the images at https://github.com/rocker-org/rocker-versioned2.
`aws_batch_job_queue`	Character of length 1, name of the AWS Batch job queue to use. There is no default for this argument, and a job queue must be created prior to running the controller. Please see https://docs.aws.amazon.com/batch/ for details.
`aws_batch_share_identifier`	`NULL` or character of length 1. For details, visit https://www.paws-r-sdk.com/docs/batch_submit_job/ and the "AWS arguments" sections of this help file.
`aws_batch_scheduling_priority_override`	`NULL` or integer of length 1. For details, visit https://www.paws-r-sdk.com/docs/batch_submit_job/ and the "AWS arguments" sections of this help file.
`aws_batch_parameters`	`NULL` or a nonempty list. For details, visit https://www.paws-r-sdk.com/docs/batch_submit_job/ and the "AWS arguments" sections of this help file.
`aws_batch_container_overrides`	`NULL` or a nonempty named list of fields to override in the container specified in the job definition. Any overrides for the `command` field are ignored because `crew.aws.batch` needs to override the command to run the `crew` worker. For more details, visit https://www.paws-r-sdk.com/docs/batch_submit_job/ and the "AWS arguments" sections of this help file.
`aws_batch_node_overrides`	`NULL` or a nonempty named list. For more details, visit https://www.paws-r-sdk.com/docs/batch_submit_job/ and the "AWS arguments" sections of this help file.
`aws_batch_retry_strategy`	`NULL` or a nonempty named list. For more details, visit https://www.paws-r-sdk.com/docs/batch_submit_job/ and the "AWS arguments" sections of this help file.
`aws_batch_propagate_tags`	`NULL` or a nonempty list. For more details, visit https://www.paws-r-sdk.com/docs/batch_submit_job/ and the "AWS arguments" sections of this help file.
`aws_batch_timeout`	`NULL` or a nonempty named list. For more details, visit https://www.paws-r-sdk.com/docs/batch_submit_job/ and the "AWS arguments" sections of this help file.
`aws_batch_tags`	`NULL` or a nonempty list. For more details, visit https://www.paws-r-sdk.com/docs/batch_submit_job/ and the "AWS arguments" sections of this help file.
`aws_batch_eks_properties_override`	`NULL` or a nonempty named list. For more details, visit https://www.paws-r-sdk.com/docs/batch_submit_job/ and the "AWS arguments" sections of this help file.

IAM policies

In order for the AWS Batch crew plugin to function properly, your IAM policy needs permission to perform the SubmitJob and TerminateJob AWS Batch API calls. For more information on AWS policies and permissions, please visit https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html.

AWS arguments

The AWS Batch controller and launcher accept many arguments which start with "aws_batch_". These arguments are AWS-Batch-specific parameters forwarded directly to the submit_job() method for the Batch client in the paws.compute R package

For a full description of each argument, including its meaning and format, please visit https://www.paws-r-sdk.com/docs/batch_submit_job/. The upstream API documentation is at https://docs.aws.amazon.com/batch/latest/APIReference/API_SubmitJob.html and the analogous CLI documentation is at https://docs.aws.amazon.com/cli/latest/reference/batch/submit-job.html.

The actual argument names may vary slightly, depending on which : for example, the aws_batch_job_definition argument of the crew AWS Batch launcher/controller corresponds to the jobDefinition argument of the web API and paws.compute::batch()$submit_job(), and both correspond to the --job-definition argument of the CLI.

Verbosity

Control verbosity with the paws.log_level global option in R. Set to 0 for minimum verbosity and 3 for maximum verbosity.

Examples

if (identical(Sys.getenv("CREW_EXAMPLES"), "true")) {
controller <- crew_controller_aws_batch(
  aws_batch_job_definition = "YOUR_JOB_DEFINITION_NAME",
  aws_batch_job_queue = "YOUR_JOB_QUEUE_NAME"
)
controller$start()
controller$push(name = "task", command = sqrt(4))
controller$wait()
controller$pop()$result
controller$terminate()
}

[Package crew.aws.batch version 0.0.6 Index]