---
title: "Your first capsule: a published table, analysed and recorded"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Your first capsule: a published table, analysed and recorded}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
# the vignette runs in full in GitHub CI and for the package tarball (NOT_CRAN), on r-universe
# (MY_UNIVERSE), on the package website (IN_PKGDOWN) and in an interactive session; when CRAN
# re-builds it in its check, none of these is set and only this chunk runs
if (!interactive() && !any(nzchar(Sys.getenv(c("NOT_CRAN", "MY_UNIVERSE", "IN_PKGDOWN"))))) {
  knitr::knit_exit()
}
knitr::opts_chunk$set(collapse = TRUE, comment = "#>")
```

Most published administrative data arrives as a table of counts by
period and group. The questions are always the same: what changed, how
sure can we be, is there a trend, and is this release the one we had
before. This vignette answers them for one real table in twenty lines,
then records the run so someone else can repeat it.

## The table

Ontario publishes the number of distinct individuals held in restrictive
confinement each fiscal year, by age category and gender. A tidy copy of
the published figures ships with the package.

```{r}
library(rmoriebricklayer)
otis <- read.csv(system.file("extdata", "otis_a01_individuals.csv",
                             package = "rmoriebricklayer"))
otis
```

## One call

```{r}
a <- anltab(otis, value = "individuals", prd = "year",
                   by = c("table", "group"))
a
```

The change table carries an exact interval for every percent change and
a p-value that is exact too. Five groups are compared twice each, so the
p-values are adjusted over the scan (Benjamini-Hochberg) before any row
is called significant. A row that looks striking on its own is judged in
the company of the other nine.

```{r}
a$change[, c("table", "group", "year", "value", "previous",
             "pctchg", "pctlo", "pctup", "padj",
             "significant")]
```

## What the release withheld

Published counts are often rounded. If this table had been rounded to
the nearest five, each cell could have come from any count within two
and a half of it, and the percent change inherits that. `rounding = 5`
adds the envelope and a combined interval that is the union of the
sampling interval and the envelope: the honest range for a number read
off a rounded table. Suppressed cells (`"x"`, `"<5"`) are handled the
same way through `suplim`.

```{r}
b <- anltab(otis, value = "individuals", prd = "year",
                   by = c("table", "group"), rnd = 5)
b$change[!is.na(b$change$previous),
         c("group", "year", "pctchg", "pctlo", "pctup",
           "compctlo", "compctup")]
```

## Trend

Three fiscal years is a short series. The Mann-Kendall test is exact for
it, and its smallest possible p-value with three points is one third, so
no trend can be declared from three years; the table says so instead of
hiding it. The Poisson rate ratio per year is still informative as a
description.

```{r}
a$trend
```

## Has the data changed since last time?

Keep the previous release and pass it as `prior`; every column is
screened. Before trusting a drift verdict, ask how often the screens fire
on data that has not changed: `drftcal()` splits one release into
random halves and reports the false-alarm rate per column.

```{r}
drftcal(otis, n = 10)
```

## Stock and flow

A count of people in confinement on a day is a stock; the people who pass
through in a year are a flow. Lakner's decomposition ties them exactly:
average daily population is person-days over the days in the period,
average length of stay is person-days over the people served, and the
identity `adp = admissions * alos / t` means a change in days must
multiply out of a change in people and a change in stay. `stkflow()`
reports both sides and checks the identity.

```{r}
sf <- stkflow(days = c(21900, 23725, 20440), people = c(300, 325, 280),
                 prd = c(2022, 2023, 2024), t = 365)
sf
```

A population that rose while stays shortened, or fell while stays
lengthened, reads differently from one where both moved together;
`adp()`, `alos()` and `admits()` give the single measures.

## Stock and flow

A count of people in confinement on a day is a stock; the people who pass
through in a year are a flow. Lakner's decomposition ties them exactly:
average daily population is person-days over the days in the period,
average length of stay is person-days over the people served, and the
identity `adp = admissions * alos / t` means a change in days must
multiply out of a change in people and a change in stay. `stkflow()`
reports both sides and checks the identity.

```{r}
sf <- stkflow(days = c(21900, 23725, 20440), people = c(300, 325, 280),
                 prd = c(2022, 2023, 2024), t = 365)
sf
```

A population that rose while stays shortened, or fell while stays
lengthened, reads differently from one where both moved together;
`adp()`, `alos()` and `admits()` give the single measures.

## Other published tables

The same approach works on any table with an estimate and its interval. The package also bundles
Statistics Canada estimates of First Nations, Metis and Inuit health and mental health
(`?indiex`). `vignette("first-nations-metis-inuit-health")` analyses them: by
distinction, with suppressed cells never imputed, and with wellness reported beside distress.

## Record it

`rptanl()` writes the whole analysis as Markdown or a single
HTML file, and `usecpstmpl()` writes a folder with the
provenance file, an `analysis.R` that pins the source bytes and runs the
steps above, and a README. The template runs as written against this
table.

```{r}
d <- usecpstmpl(tempfile("capsule-"), ex = TRUE)
list.files(d)
cat(readLines(file.path(d, "analysis.R"))[1:12], sep = "\n")
```

From here, replace the source URL and the column names in
`data_provenance.json`, run `Rscript analysis.R`, and copy the SHA-256
it prints back into the provenance file. Later runs refuse to proceed if
the bytes change, and `mnfrcmp()` compares two runs number by
number.

## The analysis in a picture

`plot()` draws the analysis of the published table and the stock-and-flow result.

```{r getting-started-getting-started-plots, fig.width = 7, fig.height = 4.5}
plot(a)
plot(sf)
```