Create task from call

Create a task based on a function call. This is fairly similar to callr::r, and forms the basis of lapply()-like task submission. Sending a call may have slightly different semantics than you expect if you send a closure (a function that binds data), and we may change behaviour here until we find a happy set of compromises. See Details for more on this. The expression task_create_call(f, list(a, b, c)) is similar to task_create_expr(f(a, b, c)), use whichever you prefer.

Usage

task_create_call(
  fn,
  args,
  environment = "default",
  driver = NULL,
  resources = NULL,
  envvars = NULL,
  parallel = NULL,
  root = NULL
)

Arguments

fn

The function to call.

args

A list of arguments to pass to the function

environment

Name of the hipercow environment to evaluate the task within.

driver

Name of the driver to use to submit the task. The default (NULL) depends on your configured drivers; if you have no drivers configured no submission happens (or indeed is possible). If you have exactly one driver configured we'll submit your task with it. If you have more than one driver configured, then we will error, though in future versions we may fall back on a default driver if you have one configured. If you pass FALSE here, submission is prevented even if you have no driver configured.

resources

A list generated by hipercow_resources giving the cluster resource requirements to run your task.

envvars

Environment variables as generated by hipercow_envvars, which you might use to control your task. These will be combined with the default environment variables (see vignettes("details"), this can be overridden by the option hipercow.default_envvars), and any driver-specific environment variables (see vignette("windows")). Variables provided here have the highest precedence. You can unset an environment variable by setting it to NA.

parallel

Parallel configuration as generated by hipercow_parallel, which defines which method, if any, will be used to initialise your task for parallel execution.

root

A hipercow root, or path to it. If NULL we search up your directory tree.

Value

A task id, a string of hex characters. Use this to interact with the task.

Details

Things are pretty unambiguous when you pass in a function from a package, especially when you refer to that package with its namespace (e.g. pkg::fn).

If you pass in the name without a namespace from a package that you have loaded with library() locally but you have not loaded with library within your hipercow environment, we may not do the right thing and you may see your task fail, or find a different function with the same name. We may change the semantics here in a future version to attach your package immediately before running the task.

If you pass in an anonymous function (e.g., function(x) x + 1) we may or may not do the right thing with respect to environment capture. We never capture the global environment so if your function is a closure that tries to bind a symbol from the global environment it will not work. Like with callr::r, anonymous functions will be easiest to think about where they are fully self contained (i.e., all inputs to the functions come through args). If you have bound a local environment, we may do slightly better, but semantics here are undefined and subject to change.

R does some fancy things with function calls that we don't try to replicate. In particular you may have noticed that this works:

c <- "x"
c(c, c) # a vector of two "x"'s

You can end up in this situation locally with:

f <- function(x) x + 1
local({
  f <- 1
  f(f) # 2
})

this is because when R looks for the symbol for the call it skips over non-function objects. We don't reconstruct environment chains in exactly the same way as you would have locally so this is not possible.

Examples

cleanup <- hipercow_example_helper()
#> ℹ This example uses a special helper

# Similar to the example in task_create_call
id <- task_create_call(stats::runif, list(5))
#> ✔ Submitted task '50aef0f6b457c3f1eda7720b3461b661' using 'example'
task_info(id)
#> 
#> ── task 50aef0f6b457c3f1eda7720b3461b661 (submitted) ───────────────────────────
#> ℹ Submitted with 'example'
#> ℹ Task type: call
#>   • Call: stats::runif
#>   • Args: 5
#>   • Environment: default
#>     R_GC_MEM_GROW: 3
#> ℹ Created at 2024-10-08 15:27:03.747485 (moments ago)
#> ! Not started yet (waiting for 74ms)
#> ! Not finished yet (waiting to start)
task_wait(id)
#> [1] TRUE
task_result(id)
#> [1] 0.89738141 0.66562014 0.06597188 0.90216301 0.89406622

# Unlike task_create_explicit, variables are automatically included:
id <- task_create_call(function(x, y) x + y, list(2, 5))
#> ✔ Submitted task 'db4615f88a02f3617c79500a5899b02b' using 'example'
task_info(id)
#> 
#> ── task db4615f88a02f3617c79500a5899b02b (submitted) ───────────────────────────
#> ℹ Submitted with 'example'
#> ℹ Task type: call
#>   • Call: (anonymous)
#>   • Args: 2, 5
#>   • Environment: default
#>     R_GC_MEM_GROW: 3
#> ℹ Created at 2024-10-08 15:27:04.83511 (moments ago)
#> ℹ Started at 2024-10-08 15:27:05.040554 (moments ago; waited 206ms)
#> ! Not finished yet (waiting to start)
task_wait(id)
#> [1] TRUE
task_result(id)
#> [1] 7

cleanup()
#> ℹ Cleaning up example