Data capsules: pin, verify, sign and re-check a CSV

A capsule is a directory holding a data file and data_provenance.json, the record of where the file came from and what it was: its SHA-256, size and row count. The record is what lets someone else (or you, a year later) confirm that an analysis still runs on the data it was written for. The morie_capsule_*() functions are thin wrappers over rmoriebricklayer; the same capsules are read by morie in Python and checked in CI by the rmbl-action GitHub Action.

library(rmorie)
dir <- file.path(tempdir(), "capsule-vignette")
dir.create(dir, showWarnings = FALSE)
set.seed(1)
utils::write.csv(data.frame(region = sample(c("east", "west"), 200, TRUE),
                            rate = stats::rnorm(200, 10, 2)),
                 file.path(dir, "rates.csv"), row.names = FALSE)

Write the record

prov <- morie_capsule_write(dir, "rates.csv", url = "https://example.org/rates.csv",
                            name = "rates", retrieved = "2026-10-20")
str(prov$resource)
#> List of 4
#>  $ filename           : chr "rates.csv"
#>  $ sha256             : chr "987f3e69f784f6cb1aa1e9bc570d56bd43da748a8b97802baaacd91463b17fdd"
#>  $ size_bytes         : num 4801
#>  $ row_count_data_rows: int 200

Verify it

Every check is a row; ok is TRUE only when all of them pass.

morie_capsule_verify(dir)
#> Capsule verification: intact (7 of 7 checks passed)
#>   ok    provenance_readable  /tmp/RtmptC55W1/capsule-vignette/data_provenance.json
#>   ok    data_present         /tmp/RtmptC55W1/capsule-vignette/rates.csv
#>   ok    data_sha256          987f3e69f784f6cb1aa1e9bc570d56bd43da748a8b97802baaacd91463b17fdd
#>   ok    data_size_bytes      4801 bytes on disk
#>   ok    data_readable        /tmp/RtmptC55W1/capsule-vignette/rates.csv
#>   ok    data_row_count       200 rows on disk
#>   ok    data_not_synthetic   no synthetic marker: sidecar, manifest and provenance all say real data

Record the results, sign and bundle

The manifest records the numbers the analysis produced next to the numbers they should reproduce. Signing it, or bundling the whole directory, shows who vouched for the capsule and that nothing changed since.

d <- utils::read.csv(file.path(dir, "rates.csv"))
man <- suppressMessages(morie_capsule_manifest(
  list(project = "rates"),
  data.frame(name = c("n", "mean_rate"), observed = c(nrow(d), mean(d$rate)),
             expected = c(200, mean(d$rate))),
  path = file.path(dir, "manifest.json")
))
key <- rmoriebricklayer::fipskg("ML-DSA-44")
sig <- morie_capsule_sign(man, key)
rmoriebricklayer::cpsvrf(rmoriebricklayer::mnfdgst(man), sig, key)
#> [1] TRUE
bundle <- morie_capsule_bundle(dir, man, key)
rmoriebricklayer::cpsbndlvrf(attr(bundle, "path"), dir, man = man)$ok
#> [1] TRUE
morie_capsule_compare(man, d, list(n = nrow, mean_rate = function(z) mean(z$rate)))$ok
#> [1] TRUE

Check a fresh download for drift

A re-released file has a new digest even when it holds the same data; morie_capsule_drift() asks whether the distributions moved.

fresh <- d
fresh$rate <- fresh$rate * 1.5
drift <- morie_capsule_drift(dir, fresh)
drift$anydrft
#> [1] TRUE
drift$columns[, c("column", "type", "p_value", "drifted")]
#>   column        type      p_value drifted
#> 1 region categorical 1.000000e+00   FALSE
#> 2   rate     numeric 1.876628e-45    TRUE