Site files & data

Pillar 1 — building, calibrating and publishing site files

Note

This is a maintainer-facing page, and one of the three maintainer pillars. If you just want to download and use a site file, see the Accessing site-files vignette and tutorial part 3.

A site file is the standardised, per-country bundle of input data that parameterises malariasimulation for a real place. This pillar is everything that has to happen for one to exist and be downloadable: collating raw global data, assembling it per country, calibrating transmission intensity against observed prevalence, and publishing the result as a versioned packet.

When a user calls site::fetch_site(), they are downloading the output of this pipeline.

What you can and cannot run yourself

The code is public. The input data is not.

The pipeline lives in a public repository, mrc-ide/foresite-orderly, with a thorough maintainer README. It is genuinely readable, and worth reading. But a fresh clone will not run:

  • The raw data and boundary files are gitignored and absent from a clone. You get the code and none of the inputs.
  • The GADM boundary files under src/data_boundaries/boundaries/<boundary>/ are large, hand-placed inputs. They are not regenerated by the pipeline and are not recoverable from git. They also determine which countries a run covers.
  • The only fully-populated copy lives on the Imperial malaria network share (\\projects.dide.ic.ac.uk\malaria, mapped as P:).

If you are outside Imperial: read the pipeline to understand provenance — what feeds what, how calibration works, what the numbers mean — but download site files with fetch_site() rather than trying to rebuild them. Published packets exist precisely so that nobody needs this infrastructure.

If you are a new Imperial maintainer: work from the populated copy on the share, not from a fresh clone, or you will lose time to a run that cannot start.

The pipeline at a glance

Everything is orderly reports, orchestrated by a single top-level script, mission_control.R. The broad shape:

flowchart LR
  raw["Raw global data"] --> proc["Processed, per country"]
  proc --> elem["Site-file elements"]
  elem --> site["Site file"]
  site --> cal["Calibrated site file"]
  cal --> rel["Diagnostics & release"]

The main build path is spatial → population → site_file → calibration → diagnostics/stats, with a fan of data_* reports supplying inputs. Two steps — demography and calibration — are dispatched to the DIDE Windows cluster via hipercow; everything else runs locally. A full local run is impractical because of those two.

The foresite-orderly README is the authority on the detail — the dependency graph, the per-report breakdown, the run sequence, and HPC setup. It is deliberately not duplicated here.

Where the data-source catalogue lives

What each dataset is, where it came from, its version and access date is maintained on the user-facing site data sources page — not in the pipeline repo and not here. If you add or change a data source, that page is the one that has to be updated, because it is what users cite.

Running the pipeline Imperial only

The canonical run is a top-to-bottom execution of mission_control.R, which defines the ISO list, configures the cluster, runs the global data layer, dispatches demography, sets the run options (boundary, admin, urban_rural, version), loops the per-country build, dispatches calibration, and finishes with diagnostics and cross-country stats.

Re-running a single country is just calling the relevant orderly_run with that ISO’s parameters — dependencies are parameter-matched, so only that country’s chain rebuilds, provided its upstream packets are already in the archive.

You need a DIDE cluster account and the project on the malaria share, so the cluster nodes can see the orderly root. See the README’s prerequisites section for the full list.

Do not clean data_boundaries

The boundary files are hand-placed and unrecoverable from git, but orderly’s cleanup scan cannot see that they are load-bearing — it treats them as deletable. A manual orderly_cleanup("data_boundaries") would wipe the boundary set and break every downstream per-country report. Preview with orderly_cleanup_status() or dry_run = TRUE first.

Calibration

calibration is the only step that runs the transmission model, and it is what turns an assembled site file into a usable one. It fills in the eir placeholders left by site_file.

The baseline EIR for each site and species is tuned so that modelled prevalence matches the Malaria Atlas Project target — PfPR2-10 for P. falciparum, PvPR for P. vivax — averaged over 2010–2024. The search uses cali::calibrate(), reading prevalence back through postie::drop_burnin() and postie::get_prevalence(). National modelled cases and deaths are then compared against the WHO World Malaria Report to derive the bias corrections stored in site$bias_correction.

The output, calibrated_scaled_site.rds, is the packet that gets released.

Calibration is contingent on the whole history

The EIR is fitted given the historical scale-up of interventions and everything else in the site file that determines past dynamics. Change any of that history and the EIR is no longer the right one. This is the same fact that users meet from the other side in tutorial part 3 — it is why simulations must cover the full period plus a burn-in, and why editing a site file means recalibrating it with cali.

Because calibration depends on site, malariasimulation, cali, postie and netz all at once, it is also the most sensitive point in the whole suite to a package compatibility problem. A breaking change that slips through usually shows up here first.

Publishing a release Imperial only

Finished packets are pushed to a Packit server:

  • Server: https://malariaverse-sitefiles.packit.dide.ic.ac.uk
  • Registered in site as orderly location malariaverse-sitefiles — see site:::location_configuration(). Users can override it with the site.orderly_location option.
  • Pushed by operations/push_packit.R, using orderly_location_push.
  • Authentication: a GitHub PAT with read:org scope, in .Renviron.

Two things reliably catch people out:

  1. Pushes are per ISO, not a bulk sync. Each push sends one calibration packet plus its entire dependency tree, including large rasters.
  2. The version string is hard-coded in push_packit.R (and in extract_files.R). It does not follow the value set in mission_control.R automatically — you have to update it by hand to the release you mean to push.

Record every release in release_log.csv.

Granting people access to this server is not part of this pillar — it is an ordinary support task, covered under Users & support.

Versioning site files

A site file is identified by boundary set, admin level, urban/rural split, and a version string of the form malariaverse_<MM_YYYY> (e.g. malariaverse_06_2026). fetch_site() returns the latest matching version unless the user pins one.

The same malariaverse_<MM_YYYY> convention also names the package git tags, and the two are deliberately offset: the current site-file release is supported by the packages’ current main branches, and the packages are only tagged when a new site-file release retires the old one. A site-file release is therefore the event that triggers tagging across the package repos — see how package tags relate to site-file releases.

This makes a site-file release a bigger deal than it looks from inside the pipeline: it is the moment the whole package suite has to be frozen for the world being left behind.

What is actually in a site file

site_file assembles a named list and validates it before saving — every site must be able to build a valid malariasimulation parameter set, interventions must not be NA, and population-at-risk must not exceed population. The elements cover: identifiers and metadata; the sites table and shape polygons; cases_deaths and prevalence; nested interventions (treatment, ITN, IRS, SMC, PMC, vaccine, LSM); population and demography; vectors and pyrethroid resistance; seasonality; blood_disorders; accessibility; and eir.

The site package is the authority on this structure and on how it becomes model input — if the two ever disagree, site is right and this page is stale.