Create a queue object. This is an R6::R6Class object which you interact with by calling "methods" which are described below, and on the help page for queuer::queue_base, from which this derives.
queue_didehpc(
context,
config = didehpc_config(),
root = NULL,
initialise = TRUE,
provision = NULL,
login = NULL
)A context
Optional dide configuration information.
A root directory, not usually needed
Passed through to the base queue. If you set
this to FALSE you will not be able to submit tasks. By default
if FALSE this also sets provision to later and login to
FALSE.
A provisioning strategy to use. Options are
verylazy (the default) which installs packages if any declared
package is not present, or if the remote library has already
been provisioned. This is lazier than the lazy policy and
faster as it avoids fetching package metadata, which may take a
few seconds. If you have manually adjusted your library
(especially by removing packages) you will probably want to use
lazy or upgrade to account for dependencies of your declared
packages.
lazy: which tells pkgdepends to be "lazy" - this prefers to
minimise installation time and does not upgrade packages unless required.
upgrade: which tells pkgdepends to always try and upgrade
later: don't do anything on creation
fake: don't do anything but mark the queue as being already
provisioned (this option can come in useful if you really don't
want to risk any accidental package installation)
Logical, indicating if we should immediately
login. If TRUE, then you will be prompted to login
immediately, rather than when a request to the web portal is
made.
queuer::queue_base -> queue_didehpc
configYour didehpc_config() for this queue.
Do not change this after queue creation as changes may not
take effect as expected.
clientA web_client object used to communicate with the web portal. See the help page for its documentation, but you will typically not need to interact with this.
Inherited methods
queuer::queue_base$enqueue()queuer::queue_base$enqueue_()queuer::queue_base$enqueue_bulk()queuer::queue_base$initialize_context()queuer::queue_base$lapply()queuer::queue_base$mapply()queuer::queue_base$task_bundle_get()queuer::queue_base$task_bundle_info()queuer::queue_base$task_bundle_list()queuer::queue_base$task_bundle_retry_failed()queuer::queue_base$task_delete()queuer::queue_base$task_get()queuer::queue_base$task_list()queuer::queue_base$task_result()queuer::queue_base$task_retry_failed()queuer::queue_base$task_status()queuer::queue_base$task_times()
new()Constructor
queue_didehpc_$new(
context,
config,
root,
initialise,
provision,
login,
client = NULL
)context, config, root, initialise, provision, loginSee above
clientA web_client object, primarily useful for testing the package
login()Log onto the web portal. This will be called automatically at either when creating the object (by default) or when you make your first request to the portal. However, you can call this to refresh the session too.
cluster_load()Report on the overall cluster usage
reconcile()Attempt to reconcile any differences in task state
between our database and the HPC queue. This is needed when
tasks have crashed, or something otherwise bad has happened
and you have tasks stuck in PENDING or RUNNING that are
clearly not happy. This function does not play well with workers and
you should not use it if using them.
submit()Submit a task to the queue. Ordinarily you do not call
this directly, it is called by the $enqueue() method of
queuer::queue_base when you create a task. However, you can
use this to resubmit a task that has failed if you think it will
run successfully a second time (e.g., because you cancelled it
the first time around).
submit_workers()Submit workers to the queue. This only works if
use_rrq or use_workers is TRUE in your configuration.
See vignette("workers") for more information.
stop_workers()Stop workers running on the cluster. See
vignette("workers") for more information. By default
workers will timeout after 10 minutes of inactivity.
rrq_controller()Return an rrq::rrq_controller object, if you have
set use_rrq or use_workers in your configuration. This is
a lightweight queue using your workers which is typically much
faster than submitting via $enqueue(). See vignette("workers")
for more information.
unsubmit()Unsubmit tasks from the cluster. This removes the tasks from the queue if they have not been started yet, and stops them if currently running. It will have no effect if the tasks are completed (successfully or errored)
dide_id()Find the DIDE task id of your task. This is the number displayed in the web portal.
dide_log()Return the pre-context log of a task. Use this to find
out what has happened to a task that has unexpectedly failed, but
for which $log() is uninformative.
provision_context()Provision your context for running on the cluster.
This sets up the remote set of packages that your tasks will use.
See vignette("packages") for more information.
queue_didehpc_$provision_context(
policy = "verylazy",
dryrun = FALSE,
quiet = FALSE,
show_progress = NULL,
show_log = TRUE
)policyThe installation policy to use, as interpreted by
pkgdepends::pkg_solution - so this should be verylazy/lazy
(install missing packages but don't upgrade unless needed) or
upgrade (upgrade packages as possible). In addition you can
also use later which does nothing, or fake which pretends
that it ran the provisioning. See vignette("packages") for
details on these options.
dryrunDo a dry run installation locally - this just checks that your requested set of packages is plausible, but does this without submitting a cluster job so it may be faster.
quietLogical, controls printing of informative messages
show_progressLogical, controls printing of a spinning progress bar
show_logLogical, controls printing of the log from the cluster
install_packages()Install packages on the cluster. This can be used to
more directly install packages on the cluster than the
$provision_context method that you would typically use.
See vignette("packages") for more information.
queue_didehpc_$install_packages(
packages,
repos = NULL,
policy = "lazy",
dryrun = FALSE,
show_progress = NULL,
show_log = TRUE
)packagesA character vector of packages to install. These
can be names of CRAN packages or GitHub references etc; see
pkgdepends::new_pkg_installation_proposal() and
vignette("packages") for more details
reposA character vector of repositories to use when installing. A suitable CRAN repo will be added if not detected.
policyThe installation policy to use, as interpreted by
pkgdepends::pkg_solution - so this should be lazy
(install missing packages but don't upgrade unless needed) or
upgrade (upgrade packages as possible). In addition you can
also use later which does nothing, or fake which pretends
that it ran the provisioning. See vignette("packages") for
details on these options.
dryrunDo a dry run installation locally - this just checks that your requested set of packages is plausible, but does this without submitting a cluster job so it may be faster.
show_progressLogical, controls printing of a spinning progress bar
show_logLogical, controls printing of the log from the cluster