Packages and provisioning

Depending on where they come from and how specific you need to be on versioning, getting packages available for your cluster tasks ranges from very straightforward to a constant struggle.

We install packages into a special directory within hipercow/ by running a special job on the cluster itself using our conan2 package. This aims to solve most of the common situations that we have seen, and has a couple of escape hatches that we hope will help with future problems.

There are three broad approaches to installing packages that we support:

Dependency resolution and package installation using pkgdepends (method = "pkgdepends")
A totally automatic method that uses pkgdepends but tries to just work out everything itself (method = "auto")
A bespoke script that you write yourself, and we run (method = "script")
Reproducible project environments with renv (method = "renv")

We expect that packages might come from any number of places:

CRAN or Bioconductor
A CRAN-like repository (such as an “R universe” or a drat repository)
A public GitHub repository, which one might ordinarily install via remotes::install_github()
Local sources, available on your computer and in the hipercow directory

We don’t currently support packages from private repositories, but let us know if that is a problem for you. We have a plan to support this later in 2024.

This all ends up being a lot of flexibility, so our (current) suggestion if you are starting out is to use the pkgdepends method with the pkgdepends.txt configuration a starting point as this will be the easiest for us to replicate issues with, and should end up being the most predictable.

library(hipercow)

Using `pkgdepends`

This is the method that is least prescriptive on your workflow and simplest. We recommend this as a good starting place.

There are two places that we might get lists of packages to install:

From a file pkgdepends.txt in the directory that also contains the hipercow/ directory
Manually, within a call to hipercow_provision()

We’ll consider these in turn here.

A list of packages

Create a file called pkgdepends.txt at the top of the hipercow root. This file has a very simple format, with each line being the name of a package, or a pkgdepends “package reference”, plus support for comments, blank lines and specifying additional repositories to install from.

So a complex case might look like

# Also use the ncov-ic universe
repo::https://ncov-ic.r-universe.dev

# Specific version of malaria simulation, via a tag
mrc-ide/malariasimulation@v1.6.0

# Package from CRAN
coda

Blank lines are ignored
Lines starting with # are comments and are also ignored
Entries are separated by newlines
Lines like coda represent packages from CRAN or another CRAN-like repository
Lines like mrc-ide/malariasimulation@v1.6.0 are a pkgdepends reference and can be quite complex to control what version is installed
Lines like repo::https://ncov-ic.r-universe.dev add additional CRAN-like repositories (this is the recommended way of installing some of our packages such as odin or orderly2)
You can install packages that you have a .tar.gz for by adding a line like local::./mypkg_0.1.0.tar.gz where the path is relative to your hipercow root folder.

The pkgdepends docs provide lots of nice examples on package references. Note that the repo::<url> line is an extension to this.

We have an new empty hipercow root, containing a simple pkgdepends.txt file:

#> .
#> ├── hipercow
#> └── pkgdepends.txt

The pkgdepends.txt file contains simply:

cowsay

We can provision this environment by running

hipercow_provision()
#> ✔ Selected provisioning method 'pkgdepends'
#> /`-'\  _______  ___  ___ ____
#> \,T./ / __/ _ \/ _ \/ _ `/ _ \
#>   |   \__/\___/_//_/\_,_/_//_/
#>   |   ---- THE  LIBRARIAN ----
#> 
#> Bootstrapping from: /home/rfitzjoh/lib/R/library
#> Installing into library: hipercow/lib
#> Using method pkgdepends
#> Running in path: /tmp/Rtmp0FgkdQ/file1af6fc77685796
#> Library paths:
#>   - /tmp/Rtmp0FgkdQ/file1af6fc77685796/hipercow/lib
#>   - /usr/local/lib/R/site-library
#>   - /usr/lib/R/site-library
#>   - /usr/lib/R/library
#> id: 20240126113025
#> Logs from pkgdepends follow:
#> 
#> -------------------------------------------------------------------------------
#> 
#> 
#> ── repos 
#> • https://cloud.r-project.org
#> 
#> ── refs 
#> • cowsay
#> 
#> ✔ Updated metadata database: 4.19 MB in 9 files.
#> 
#> ℹ Updating metadata database
#> ✔ Updating metadata database ... done
#> 
#> + cowsay     0.9.0 [bld][dl]
#> + crayon     1.5.2 [bld][dl]
#> + fortunes   1.5-4 [bld][dl]
#> + rmsfact    0.0.3 [bld][dl]
#> ℹ Getting 4 pkgs with unknown sizes
#> ✔ Got rmsfact 0.0.3 (source) (10.73 kB)
#> ✔ Got fortunes 1.5-4 (source) (192.94 kB)
#> ✔ Got crayon 1.5.2 (source) (40.57 kB)
#> ✔ Got cowsay 0.9.0 (source) (573.56 kB)
#> ℹ Building crayon 1.5.2
#> ℹ Building fortunes 1.5-4
#> ℹ Building rmsfact 0.0.3
#> ✔ Built rmsfact 0.0.3 (883ms)
#> ✔ Built fortunes 1.5-4 (1s)
#> ✔ Installed rmsfact 0.0.3  (46ms)
#> ✔ Installed fortunes 1.5-4  (29ms)
#> ✔ Built crayon 1.5.2 (1.6s)
#> ✔ Installed crayon 1.5.2  (12ms)
#> ℹ Building cowsay 0.9.0
#> ✔ Built cowsay 0.9.0 (806ms)
#> ✔ Installed cowsay 0.9.0  (17ms)
#> ✔ Summary:   4 new  in 4.4s
#> 
#> -------------------------------------------------------------------------------
#> Writing library description to 'hipercow/lib/.conan/20240126113025'
#> Done!

This launches a small task on the cluster and uses pkgdepends to install cowsay and all its dependencies. If recent installations have been attempted, it will use cached copies, which should make things fairly snappy.

Once installed we can run a task that uses this package:

id <- task_create_expr(cowsay::say("HiperMoo", "cow"))
#> ✔ Submitted task '1d520a4efa59c757917b2ad74148a48c' using 'example'
task_wait(id)
#> [1] TRUE
task_log_show(id)
#> 
#> ── hipercow 0.2.33 running at '/tmp/Rtmp0FgkdQ/file1af6fc77685796' ─────────────
#> ℹ library paths:
#> • /home/rfitzjoh/lib/R/library
#> • /usr/local/lib/R/site-library
#> • /usr/lib/R/site-library
#> • /usr/lib/R/library
#> ℹ id: 1d520a4efa59c757917b2ad74148a48c
#> ℹ starting at: 2024-01-26 11:30:34.348013
#> ℹ task type: expression
#> ℹ expression: cowsay::say("HiperMoo", "cow")
#> ℹ no local variables
#> ───────────────────────────────────────────────────────────────── task logs ↓ ──
#> 
#>  ----- 
#> HiperMoo 
#>  ------ 
#>     \   ^__^ 
#>      \  (oo)\ ________ 
#>         (__)\         )\ /\ 
#>              ||------w|
#>              ||      ||
#> ───────────────────────────────────────────────────────────────── task logs ↑ ──
#> ✔ status: success
#> ℹ finishing at: 2024-01-26 11:30:34.348013 (elapsed: 0.1545 secs)

For all methods of provisioning you can see what has been done previously by running hipercow_provision_list() and hipercow_provision_check(). Running hipercow_provision_list() just lists the installations into your library:

hipercow_provision_list()
#> ℹ 1 conan installation recorded
#> • 1: 20240126113025 (moments ago) [0]

The hipercow_provision_check() function also compares previously performed installations with the what hipercow_provision() would do with the same arguments:

hipercow_provision_check()
#> ✔ Selected provisioning method 'pkgdepends'
#> ℹ 1 conan installation recorded
#> • 1: 20240126113025 (moments ago) [0] (*)
#> ℹ The entry marked with '*' matches the provided installation hash

You can see what is in a library by using hipercow_provision_compare(); this compares between two versions of the library. Because we only have one version here the comparison will be against the empty library:

hipercow_provision_compare()
#> ── Comparing conan installations ───────────────────────────────────────────────
#> • (empty installation)
#> • 20240126113025 1st; current installation (moments ago)
#> 
#> ── 4 added packages ──
#> 
#> • cowsay (0.9.0) CRAN
#> • crayon (1.5.2) CRAN
#> • fortunes (1.5.4) CRAN
#> • rmsfact (0.0.3) CRAN

Manually

Sometimes, regardless of how things are installed, you need to install something manually. For example, you just want to change the version being used of some package, or pkgdepends fails to resolve a nice set of dependencies from some complex interdependent set of packages and you want to install a specific version on top of everything it did.

To do this, hipercow_provision with method = "pkgdepends" and refs as a character vector of package references, following the same format as pkgdepends.txt

hipercow_provision(method = "pkgdepends", refs = "cran::coda")
#> /`-'\  _______  ___  ___ ____
#> \,T./ / __/ _ \/ _ \/ _ `/ _ \
#>   |   \__/\___/_//_/\_,_/_//_/
#>   |   ---- THE  LIBRARIAN ----
#> 
#> Bootstrapping from: /home/rfitzjoh/lib/R/library
#> Installing into library: hipercow/lib
#> Using method pkgdepends
#> Running in path: /tmp/Rtmp0FgkdQ/file1af6fc77685796
#> Library paths:
#>   - /tmp/Rtmp0FgkdQ/file1af6fc77685796/hipercow/lib
#>   - /usr/local/lib/R/site-library
#>   - /usr/lib/R/site-library
#>   - /usr/lib/R/library
#> id: 20240126113035
#> Logs from pkgdepends follow:
#> 
#> -------------------------------------------------------------------------------
#> 
#> 
#> ── repos 
#> • https://cloud.r-project.org
#> 
#> ── refs 
#> • cran::coda
#> ℹ Loading metadata database
#> ✔ Loading metadata database ... done
#> 
#> + coda   0.19-4 [bld]
#> ℹ No downloads are needed, 1 pkg is cached
#> ✔ Got coda 0.19-4 (source) (74.24 kB)
#> ℹ Building coda 0.19-4
#> ✔ Built coda 0.19-4 (1.8s)
#> ✔ Installed coda 0.19-4  (13ms)
#> ✔ Summary:   1 new   1 kept  in 1.8s
#> 
#> -------------------------------------------------------------------------------
#> Writing library description to 'hipercow/lib/.conan/20240126113035'
#> Done!

Now hipercow_provision_list() and hipercow_provision_compare()` show we have two steps in the history of the library and we can see what has changed:

hipercow_provision_list()
#> ℹ 2 conan installations recorded
#> • 1: 20240126113025 (moments ago) [-1]
#> • 2: 20240126113035 (moments ago) [0]
hipercow_provision_compare()
#> ── Comparing conan installations ───────────────────────────────────────────────
#> • 20240126113025 1st; previous installation (moments ago)
#> • 20240126113035 2nd; current installation (moments ago)
#> 
#> ── 4 unchanged packages ──
#> 
#> ℹ To show unchanged packages, print with 'show_unchanged = TRUE'
#> 
#> ── 1 added package ──
#> 
#> • coda (0.19.4) CRAN

Automatically, from an environment

The pkgdepends approach above duplicates some information already present in the packages listed in your calls to hipercow_environment(), implied in the files source()’d at the start of your task and from the local metadata about where your packages were installed from in the first place.

We try to poke around in your script files and find all the packages that you use, then we look at your installation, and from that build a set of references that we hope will recreate the installation.

We have an new empty hipercow root, this time containing some source file src.R:

#> .
#> ├── hipercow
#> └── src.R

The file src.R contains the code that we want to run on the cluster:

cowsay_fact <- function() {
  cowsay::say("catfact", "cow")
}

We tell hipercow about this:

hipercow_environment_create(sources = "src.R")
#> ✔ Created environment 'default'

Now, when we run hipercow_provision(), hipercow will check through our src.R file and seen that we use cowsay

hipercow_provision()
#> ✔ Selected provisioning method 'auto'
#> /`-'\  _______  ___  ___ ____
#> \,T./ / __/ _ \/ _ \/ _ `/ _ \
#>   |   \__/\___/_//_/\_,_/_//_/
#>   |   ---- THE  LIBRARIAN ----
#> 
#> Bootstrapping from: /home/rfitzjoh/lib/R/library
#> Installing into library: hipercow/lib
#> Using method auto
#> Running in path: /tmp/Rtmp0FgkdQ/file1af6fc1833a1ef
#> Library paths:
#>   - /tmp/Rtmp0FgkdQ/file1af6fc1833a1ef/hipercow/lib
#>   - /usr/local/lib/R/site-library
#>   - /usr/lib/R/site-library
#>   - /usr/lib/R/library
#> id: 20240126113040
#> Logs from pkgdepends follow:
#> 
#> -------------------------------------------------------------------------------
#> 
#> 
#> ── repos 
#> • https://cloud.r-project.org
#> 
#> ── refs 
#> • cowsay
#> 
#> ✔ Updated metadata database: 4.19 MB in 9 files.
#> 
#> ℹ Updating metadata database
#> ✔ Updating metadata database ... done
#> 
#> ! Failed to update system requirement mappings, will use cached mappings.
#> + cowsay     0.9.0 [bld][dl]
#> + crayon     1.5.2 [bld][dl]
#> + fortunes   1.5-4 [bld][dl]
#> + rmsfact    0.0.3 [bld][dl]
#> ℹ Getting 4 pkgs with unknown sizes
#> ✔ Got crayon 1.5.2 (source) (40.57 kB)
#> ✔ Got rmsfact 0.0.3 (source) (10.73 kB)
#> ✔ Got fortunes 1.5-4 (source) (192.94 kB)
#> ✔ Got cowsay 0.9.0 (source) (573.56 kB)
#> ℹ Building crayon 1.5.2
#> ℹ Building fortunes 1.5-4
#> ℹ Building rmsfact 0.0.3
#> ✔ Built rmsfact 0.0.3 (880ms)
#> ✔ Built fortunes 1.5-4 (985ms)
#> ✔ Installed rmsfact 0.0.3  (44ms)
#> ✔ Installed fortunes 1.5-4  (29ms)
#> ✔ Built crayon 1.5.2 (1.5s)
#> ✔ Installed crayon 1.5.2  (34ms)
#> ℹ Building cowsay 0.9.0
#> ✔ Built cowsay 0.9.0 (803ms)
#> ✔ Installed cowsay 0.9.0  (17ms)
#> ✔ Summary:   4 new  in 4.3s
#> 
#> -------------------------------------------------------------------------------
#> Writing library description to 'hipercow/lib/.conan/20240126113040'
#> Done!

Now our code works!

id <- task_create_expr(cowsay_fact())
#> ✔ Submitted task '90d0a213194dbc448f0af38f3b5ac31e' using 'example'
task_wait(id)
#> [1] TRUE
task_log_show(id)
#> 
#> ── hipercow 0.2.33 running at '/tmp/Rtmp0FgkdQ/file1af6fc1833a1ef' ─────────────
#> ℹ library paths:
#> • /home/rfitzjoh/lib/R/library
#> • /usr/local/lib/R/site-library
#> • /usr/lib/R/site-library
#> • /usr/lib/R/library
#> ℹ id: 90d0a213194dbc448f0af38f3b5ac31e
#> ℹ starting at: 2024-01-26 11:30:48.655369
#> ℹ task type: expression
#> ℹ expression: cowsay_fact()
#> ℹ no local variables
#> ───────────────────────────────────────────────────────────────── task logs ↓ ──
#> 
#>  -------------- 
#> Your cat's heart beats at a rate almost double that of yours, from 110-140 beats per minute. 
#>  --------------
#>     \
#>       \
#>         \
#>             |\___/|
#>           ==) ^Y^ (==
#>             \  ^  /
#>              )=*=(
#>             /     \
#>             |     |
#>            /| | | |\
#>            \| | |_|/\
#>       jgs  //_// ___/
#>                \_)
#>   
#> ───────────────────────────────────────────────────────────────── task logs ↑ ──
#> ✔ status: success
#> ℹ finishing at: 2024-01-26 11:30:48.655369 (elapsed: 0.9717 secs)

If we had installed cowsay initially by running remotes::install_github() it would have noticed this and installed the package that way. For example, the rfiglet package is not available on CRAN and must be installed via remotes:

remotes::install_github("richfitz/rfiglet")

I’ve adjusted our src.R to use rfiglet:

cowsay_fact <- function() {
  cowsay::say("catfact", "cow")
}

figlet_date <- function() {
  print(rfiglet::figlet(as.character(Sys.Date())))
}

and now when we call hipercow_provision() you can see it installs rfiglet too, automatically finding it on GitHub!

hipercow_provision()
#> ✔ Selected provisioning method 'auto'
#> /`-'\  _______  ___  ___ ____
#> \,T./ / __/ _ \/ _ \/ _ `/ _ \
#>   |   \__/\___/_//_/\_,_/_//_/
#>   |   ---- THE  LIBRARIAN ----
#> 
#> Bootstrapping from: /home/rfitzjoh/lib/R/library
#> Installing into library: hipercow/lib
#> Using method auto
#> Running in path: /tmp/Rtmp0FgkdQ/file1af6fc1833a1ef
#> Library paths:
#>   - /tmp/Rtmp0FgkdQ/file1af6fc1833a1ef/hipercow/lib
#>   - /usr/local/lib/R/site-library
#>   - /usr/lib/R/site-library
#>   - /usr/lib/R/library
#> id: 20240126113050
#> Logs from pkgdepends follow:
#> 
#> -------------------------------------------------------------------------------
#> 
#> 
#> ── repos 
#> • https://cloud.r-project.org
#> 
#> ── refs 
#> • cowsay
#> • richfitz/rfiglet@HEAD
#> ℹ Loading metadata database
#> ✔ Loading metadata database ... done
#> 
#> + rfiglet   0.2.0 [bld][cmp] (GitHub: d713c1b)
#> ℹ No downloads are needed, 1 pkg is cached
#> ✔ Got rfiglet 0.2.0 (source) (144.05 kB)
#> ℹ Packaging rfiglet 0.2.0
#> ✔ Packaged rfiglet 0.2.0 (318ms)
#> ℹ Building rfiglet 0.2.0
#> ✔ Built rfiglet 0.2.0 (979ms)
#> ✔ Installed rfiglet 0.2.0 (github::richfitz/rfiglet@d713c1b) (14ms)
#> ✔ Summary:   1 new   4 kept  in 992ms
#> 
#> -------------------------------------------------------------------------------
#> Writing library description to 'hipercow/lib/.conan/20240126113050'
#> Done!

id <- task_create_expr(figlet_date())
#> ✔ Submitted task 'eb00966e02cecd38fddb79e8b262b59d' using 'example'
task_wait(id)
#> [1] TRUE
task_log_show(id)
#> 
#> ── hipercow 0.2.33 running at '/tmp/Rtmp0FgkdQ/file1af6fc1833a1ef' ─────────────
#> ℹ library paths:
#> • /home/rfitzjoh/lib/R/library
#> • /usr/local/lib/R/site-library
#> • /usr/lib/R/site-library
#> • /usr/lib/R/library
#> ℹ id: eb00966e02cecd38fddb79e8b262b59d
#> ℹ starting at: 2024-01-26 11:30:55.72189
#> ℹ task type: expression
#> ℹ expression: figlet_date()
#> ℹ no local variables
#> ───────────────────────────────────────────────────────────────── task logs ↓ ──
#>  ____   ___ ____  _  _          ___  _      ____   __
#> |___ \ / _ \___ \| || |        / _ \/ |    |___ \ / /_
#>   __) | | | |__) | || |_ _____| | | | |_____ __) | '_ \
#>  / __/| |_| / __/|__   _|_____| |_| | |_____/ __/| (_) |
#> |_____|\___/_____|  |_|        \___/|_|    |_____|\___/
#> 
#> ───────────────────────────────────────────────────────────────── task logs ↑ ──
#> ✔ status: success
#> ℹ finishing at: 2024-01-26 11:30:55.72189 (elapsed: 0.1573 secs)

Using a script

Sometimes you just need a lot more control than pkgdepends can provide and you want to do your own thing. To support this, create a file provision.R in the root of your hipercow repository and within this write whatever you want to install packages.

Here, we have another empty hipercow root, containing a file provision.R:

#> .
#> ├── hipercow
#> └── provision.R

The provision.R file contains simply:

install.packages("cowsay")
cowsay::say("Moo", "cow")

Here, it’s very simple, and just installs the package (plus runs a simple function from it).

hipercow_provision()
#> ✔ Selected provisioning method 'script'
#> /`-'\  _______  ___  ___ ____
#> \,T./ / __/ _ \/ _ \/ _ `/ _ \
#>   |   \__/\___/_//_/\_,_/_//_/
#>   |   ---- THE  LIBRARIAN ----
#> 
#> Bootstrapping from: /home/rfitzjoh/lib/R/library
#> Installing into library: hipercow/lib
#> Using method script
#> Running in path: /tmp/Rtmp0FgkdQ/file1af6fc18c44b73
#> Library paths:
#>   - /tmp/Rtmp0FgkdQ/file1af6fc18c44b73/hipercow/lib
#>   - /usr/local/lib/R/site-library
#>   - /usr/lib/R/site-library
#>   - /usr/lib/R/library
#> id: 20240126113056
#> Logs from your installation script 'provision.R' follow:
#> 
#> -------------------------------------------------------------------------------
#> 
#> 
#> > install.packages("cowsay")
#> Installing package into ‘/tmp/Rtmp0FgkdQ/file1af6fc18c44b73/hipercow/lib’
#> (as ‘lib’ is unspecified)
#> also installing the dependencies ‘crayon’, ‘fortunes’, ‘rmsfact’
#> 
#> trying URL 'https://cloud.r-project.org/src/contrib/crayon_1.5.2.tar.gz'
#> Content type 'application/x-gzip' length 40567 bytes (39 KB)
#> ==================================================
#> downloaded 39 KB
#> 
#> trying URL 'https://cloud.r-project.org/src/contrib/fortunes_1.5-4.tar.gz'
#> Content type 'application/x-gzip' length 192938 bytes (188 KB)
#> ==================================================
#> downloaded 188 KB
#> 
#> trying URL 'https://cloud.r-project.org/src/contrib/rmsfact_0.0.3.tar.gz'
#> Content type 'application/x-gzip' length 10728 bytes (10 KB)
#> ==================================================
#> downloaded 10 KB
#> 
#> trying URL 'https://cloud.r-project.org/src/contrib/cowsay_0.9.0.tar.gz'
#> Content type 'application/x-gzip' length 573560 bytes (560 KB)
#> ==================================================
#> downloaded 560 KB
#> 
#> * installing *source* package ‘crayon’ ...
#> ** package ‘crayon’ successfully unpacked and MD5 sums checked
#> ** using staged installation
#> ** R
#> ** byte-compile and prepare package for lazy loading
#> ** help
#> *** installing help indices
#> ** building package indices
#> ** testing if installed package can be loaded from temporary location
#> ** testing if installed package can be loaded from final location
#> ** testing if installed package keeps a record of temporary installation path
#> * DONE (crayon)
#> * installing *source* package ‘fortunes’ ...
#> ** package ‘fortunes’ successfully unpacked and MD5 sums checked
#> ** using staged installation
#> ** R
#> ** inst
#> ** byte-compile and prepare package for lazy loading
#> ** help
#> *** installing help indices
#> ** building package indices
#> ** installing vignettes
#> ** testing if installed package can be loaded from temporary location
#> ** testing if installed package can be loaded from final location
#> ** testing if installed package keeps a record of temporary installation path
#> * DONE (fortunes)
#> * installing *source* package ‘rmsfact’ ...
#> ** package ‘rmsfact’ successfully unpacked and MD5 sums checked
#> ** using staged installation
#> ** R
#> ** inst
#> ** byte-compile and prepare package for lazy loading
#> ** help
#> *** installing help indices
#> ** building package indices
#> ** testing if installed package can be loaded from temporary location
#> ** testing if installed package can be loaded from final location
#> ** testing if installed package keeps a record of temporary installation path
#> * DONE (rmsfact)
#> * installing *source* package ‘cowsay’ ...
#> ** package ‘cowsay’ successfully unpacked and MD5 sums checked
#> ** using staged installation
#> ** R
#> ** inst
#> ** byte-compile and prepare package for lazy loading
#> ** help
#> *** installing help indices
#> ** building package indices
#> ** installing vignettes
#> ** testing if installed package can be loaded from temporary location
#> ** testing if installed package can be loaded from final location
#> ** testing if installed package keeps a record of temporary installation path
#> * DONE (cowsay)
#> 
#> The downloaded source packages are in
#>  ‘/tmp/RtmpDKlA7e/downloaded_packages’
#> 
#> > cowsay::say("Moo", "cow")
#> 
#>  ----- 
#> Moo 
#>  ------ 
#>     \   ^__^ 
#>      \  (oo)\ ________ 
#>         (__)\         )\ /\ 
#>              ||------w|
#>              ||      ||
#> -------------------------------------------------------------------------------
#> Writing library description to 'hipercow/lib/.conan/20240126113056'
#> Done!

Please be considerate and don’t write scripts that will take very long to run (e.g., no longer than 10-15 minutes) or they may be killed by the cluster scheduler. Please also let us know if you have a script that takes this long.

Using renv

If you are using renv to set up your packages, you can use this with hipercow, and this should be detected automatically. With your renv project loaded and a lockfile created (so that renv::status() reports “No issues found – the project is in a consistent state.”) you should be able to run

hipercow_provision()

and see renv build a new library based on your lockfile. New tasks launched after this will use that library.

The workflow differs very slightly from using renv normally in that we take some steps to prevent renv updating the library automatically while your tasks run because this will result in disaster if multiple tasks trigger this in parallel.

Some details about the process

The general hope is that relatively little configuration or specific calls are needed here, but this comes at the cost of a little magic.

If the method argument to hipercow_provision() is not given, then we use a simple heuristic to select a method:

If a file provision.R is present, we use the script method; this takes precedence because it’s the biggest, least clever hammer. You might use this as a last resort and so it should override any other method that you might be using.
If a file pkgdepends.txt is present, we use the pkgdepends method; we think that this method is a reasonable tradeoff of predictability and usability and should cover most people’s needs.
Otherwise we’ll try for an automatic installation; this is to help people get started quickly.

You can always pass the method explicitly and any arguments supported by that method (see ?hipercow_provision for details). For example, suppose you normally had things working with pkgdepends but hit trouble with some incompatible set of versions, you might write a file fix_install.R and run it with:

hipercow_provision(method = "script", script = "fix_install.R")

If you interrupt the installation while it runs, it will not (currently) cancel the running task. We may change this in the future. Please don’t submit another provisioning task until yours is finished (you can use the HPC portal to check easily).