---
title: "JSON without jsonlite: parsing, encoding and exact serialisation"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{JSON without jsonlite: parsing, encoding and exact serialisation}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
# the vignette runs in full in GitHub CI and for the package tarball (NOT_CRAN), on r-universe
# (MY_UNIVERSE), on the package website (IN_PKGDOWN) and in an interactive session; when CRAN
# re-builds it in its check, none of these is set and only this chunk runs
if (!interactive() && !any(nzchar(Sys.getenv(c("NOT_CRAN", "MY_UNIVERSE", "IN_PKGDOWN"))))) {
  knitr::knit_exit()
}
knitr::opts_chunk$set(collapse = TRUE, comment = "#>")
library(rmoriebricklayer)
```

Manifests, provenance records and signed documents are JSON. The package parses and writes
JSON natively, in R with no parser dependency, with jsonlite's conventions, so a capsule has no dependency on another
package's parser and the parser can be held to the same standard as the cryptography.

## Encoding R objects

`tojs()` follows jsonlite's `toJSON()` conventions (data frames as rows,
`auto_unbox`, `pretty`, `na` handling):

```{r json-and-serialisation-encode}
tojs(list(a = 1:3, b = "x"), auto_unbox = TRUE)
tojs(data.frame(id = 1:2, v = c(1.5, NA)), pretty = TRUE)
```

## Parsing JSON

`fromjs()` follows `fromJSON()`, with the simplification options:

```{r json-and-serialisation-parse}
txt <- '{"id":[1,2,3],"name":["a","b","c"],"ok":true}'
str(fromjs(txt))
str(fromjs(txt, simplifyVector = FALSE))
```

### A repeated key is an error

JSON permits duplicate keys and most parsers silently keep the last one, which has been the
root of several real vulnerabilities (two parsers disagreeing about which value wins). Here a
duplicate key is an error unless you ask to keep it:

```{r json-and-serialisation-dup}
try(fromjs('{"a":1,"a":2}'))
fromjs('{"a":1,"a":2}', duplicate_keys = "keep")
```

### Big integers

Numbers beyond the exact range of a double are flagged, and can be returned as text so no
digit is lost:

```{r json-and-serialisation-bigint}
fromjs('{"id": 9007199254740993}', bigint_as_char = TRUE)
```

## Exact serialisation: types and attributes included

Data JSON loses R's structure: a factor becomes strings, a matrix becomes nested arrays, a
`Date` becomes text. `jsser()` writes an object *with* its type and
attributes, so the round trip is `identical()`:

```{r json-and-serialisation-serialize}
f <- factor(c("b", "a", "b"), levels = c("a", "b", "c"))
back <- jsunser(jsser(f))
identical(back, f)

m <- matrix(1:6, nrow = 2)
identical(jsunser(jsser(m)), m)

x <- list(a = 1:3, b = list(c = "x", d = NULL), e = TRUE)
identical(jsunser(jsser(x)), x)
```

Plain data JSON does not keep either, which is the trade-off:

```{r json-and-serialisation-serialize-vs}
tojs(f)
```

Doubles are rounded to `digits` by default; `I(17)` keeps them exact to the last bit:

```{r json-and-serialisation-digits}
jsser(1 / 3)
exact <- jsser(1 / 3, digits = I(17))
identical(jsunser(exact), 1 / 3)
```

The serialised form is text, so it can be pinned like any other:

```{r json-and-serialisation-pin}
csha256(jsser(m))
```

## Compressed, base64 JSON

For embedding a table inside another document (a manifest field, a URL, a signed record),
`jsgzenc()` gzips the JSON and base64-encodes the bytes:

```{r json-and-serialisation-gzip}
x <- list(rows = data.frame(id = 1:50, value = round(stats::runif(50), 3)))
enc <- jsgzenc(x)
substring(enc, 1, 40)
c(json = nchar(tojs(x)), gzipb64 = nchar(enc))
identical(jsgzdec(enc)$rows$id, 1:50)
```

Base64 and base64url are available on their own:

```{r json-and-serialisation-b64}
jsb64enc(charToRaw("manifest"))
rawToChar(jsb64dec("bWFuaWZlc3Q="))
```

## Why the parser is held to a security standard

Every network body, cached manifest and signature file the package reads goes through this
parser. It was reviewed for the classic parser failures and each is tested: duplicate keys
(above), string assembly that was quadratic (250,000 escapes once took 16 minutes; it is
linear now), nesting depth, numbers that are not numbers, bytes that are not UTF-8, and a
body that is a bare URL, which is "not JSON", never a second fetch. The parser and the
encoder are R code (see below), so they run under R's own memory management rather than a
sanitizer; the test suite carries the corpus of malformed inputs the review produced.


## Sending a table as JSON

A published table travels well as JSON, with its quality flags kept:

```{r json-and-serialisation-indigenous-json}
d <- indiex("infantmort")
js <- tojs(d[d$period == "2014/2016", ], auto_unbox = TRUE)
substring(js, 1, 200)
back <- fromjs(js)
all.equal(back$estimate, d$estimate[d$period == "2014/2016"])
```

## The table after the round trip in a picture

These are the estimates and intervals that arrived after the trip through JSON, one row for each
group: what the receiver plots is what the sender sent.

```{r json-and-serialisation-json-roundtrip-plot, fig.width = 7, fig.height = 4.5}
o <- order(back$estimate)
t1 <- back[o, ]
yy <- seq_len(nrow(t1))
op <- par(mar = c(4, 16, 3, 1))
plot(t1$estimate, yy, xlim = range(t1$lower, t1$upper), yaxt = "n", pch = 19, col = blpal()[1],
     xlab = "estimate and 95% interval", ylab = "", main = t1$measure[1])
segments(t1$lower, yy, t1$upper, yy, col = blpal()[1])
axis(2, yy, t1$group, las = 1, cex.axis = 0.7)
par(op)
```