DIDE Cluster Support
NOTICE: This will only be of use to people at DIDE, as it uses our cluster web portal, local cluster, and local network file systems.
What is this?
This is a package for interfacing with the DIDE cluster directly from R. It is meant make jobs running on the cluster appear as if they are running locally but asynchronously. The idea is to let the cluster appear as an extension of your own computer so you can get using it within an R project easily.
How does it work?
The steps below are described in more detail in the vignettes
- Ensure that your project is in a directory that the cluster can see (i.e. on one of the network drives). See notes for instructions
- Set your DIDE credentials up so that you can log in and tell
didehpcabout them. - Create a “context” in which future expressions will be evaluated (which will be recreated on the cluster)
- Create a “queue” that uses that context
- Queue expressions which will be run at some future time on the cluster
- Monitor progress, retrieve results, etc.
Documentation
- New to this? The main vignette contains full instructions and explanations about why some bits are needed.
- Need a reminder? There is a quickstart guide which is much shorter and will be quicker to glance through.
- Trying to install packages on the cluster? Check the packages vignette for ways of controlling this.
- Having problems? Check the troubleshooting guide.
- Lots of small jobs to run? Consider using workers for a fast queue over several cluster nodes.
Issues
- Check the issue tracker for known problems, or to create a new one
- Use the “Cluster” channel on Teams, which Rich and Wes keep an eye on
Installation
The simplest approach is to run:
# install.packages("drat") # if needed
drat:::add("mrc-ide")
install.packages("didehpc")
