Create a task based on an expression — task_create

Create a task based on an expression. This is similar to task_create_explicit except more magic, and is closer to the interface that we expect people will use.

Usage

task_create_expr(
  expr,
  environment = "default",
  driver = NULL,
  resources = NULL,
  envvars = NULL,
  parallel = NULL,
  root = NULL
)

Arguments

expr: The expression, does not need quoting. See Details.
environment: Name of the hipercow environment to evaluate the task within.
driver: Name of the driver to use to submit the task. The default (NULL) depends on your configured drivers; if you have no drivers configured no submission happens (or indeed is possible). If you have exactly one driver configured we'll submit your task with it. If you have more than one driver configured, then we will error, though in future versions we may fall back on a default driver if you have one configured. If you pass FALSE here, submission is prevented even if you have no driver configured.
resources: A list generated by hipercow_resources giving the cluster resource requirements to run your task.
envvars: Environment variables as generated by hipercow_envvars, which you might use to control your task. These will be combined with the default environment variables (see vignettes("details"), this can be overridden by the option hipercow.default_envvars), and any driver-specific environment variables (see vignette("windows")). Variables provided here have the highest precedence. You can unset an environment variable by setting it to NA.
parallel: Parallel configuration as generated by hipercow_parallel, which defines which method, if any, will be used to initialise your task for parallel execution.
root: A hipercow root, or path to it. If NULL we search up your directory tree.

Value

A task id, a string of hex characters. Use this to interact with the task.

Details

The expression passed as expr will typically be a function call (e.g., f(x)). We will analyse the expression and find all variables that you reference (in the case of f(x) this is x) and combine this with the function name to run on the cluster. If x cannot be found in your calling environment we will error; this behaviour is subject to change so let us know if you have other thoughts.

Alternatively you may provide a multiline statement by using {} to surround multiple lines, such as:

task_create_expr({
  x <- runif(1)
  f(x)
}, ...)

in this case, we apply a simple heuristic to work out that x is locally assigned and should not be saved with the expression.

If you reference values that require a lot of memory, hipercow will error and refuse to save the task. This is to prevent you accidentally including values that you will make available through an environment, and to prevent making the hipercow directory excessively large. Docs on controlling this process are still to be written.

Examples

cleanup <- hipercow_example_helper()
#> ℹ This example uses a special helper

# Similar to task_create_explicit, but we don't include the 'quote'
id <- task_create_expr(runif(5))
#> ✔ Submitted task 'fe55e86f040aa57e9d6a209f4435d01e' using 'example'
task_wait(id)
#> [1] TRUE
task_result(id)
#> [1] 0.726950803 0.005630716 0.693357676 0.974422953 0.408112767

# Unlike task_create_explicit, variables are automatically included:
n <- 3
id <- task_create_expr(runif(n))
#> ✔ Submitted task 'e7223698fb9ca9cad808a4895c9ae16c' using 'example'
task_info(id)
#> 
#> ── task e7223698fb9ca9cad808a4895c9ae16c (submitted) ───────────────────────────
#> ℹ Submitted with 'example'
#> ℹ Task type: expression
#>   • Expression: runif(n)
#>   • Locals: n
#>   • Environment: default
#>     R_GC_MEM_GROW: 3
#> ℹ Created at 2024-10-08 15:27:11.198743 (moments ago)
#> ! Not started yet (waiting for 50ms)
#> ! Not finished yet (waiting to start)
task_wait(id)
#> [1] TRUE
task_result(id)
#> [1] 0.6336758 0.9043801 0.8122358

cleanup()
#> ℹ Cleaning up example