Parsing SIU director’s reports

Ontario’s Special Investigations Unit publishes a director’s report for every investigation as an HTML page. The reports share a structure but not a template: fields appear under varying headings, counts are written in words or figures, and the text is long. The package parses them into a fixed, panel-reviewed schema with a parser written for hostile input.

The schema

sch <- siusch()
head(sch$name, 20)
#>  [1] "police_service"                "date_of_incident_iso"         
#>  [3] "date_siu_notified_iso"         "date_of_director_decision_iso"
#>  [5] "siu_investigators"             "siu_forensics_investigators"  
#>  [7] "number_of_witness_officials"   "number_of_civilian_witnesses" 
#>  [9] "number_of_subject_officials"   "age_affected"                 
#> [11] "sex_gender_affected"           "charges_recommended"          
#> [13] "directors_name"                "location_of_call"             
#> [15] "specific_injuries"             "relevant_legislation"
nrow(sch)
#> [1] 16

Parsing a report

The package ships a synthetic report with the structure of a real one and no real person in it:

f <- siuprs(system.file("extdata", "siu_synthetic_report.html",
                                      package = "rmoriebricklayer"))
f[["number_of_subject_officials"]]
#> [1] "2"
f[["date_of_incident_iso"]]
#> [1] "2023-01-05"

The pieces

HTML to plain text, with the layout tags turned into line structure and entities decoded:

siutxt("<p>Number of SIU Investigators assigned: 3</p>")
#> [1] " Number of SIU Investigators assigned: 3\n"

The subject-official count, resolved from the enumerated list rather than a headline number that reports sometimes omit:

siuso("Subject Officials\nSO #1 Interviewed\nSO #2 Declined interview")
#> $count
#> [1] 2
#> 
#> $reason
#> [1] "section: max(ordinal 2, entries 2)"

Dates in the report’s prose, normalised to ISO:

siudate("March 4, 2024")
#> [1] "2024-03-04"
siudate("not a date")
#> [1] ""

Written for hostile input

A parser that reads 5,000 public web pages will meet broken HTML, megabyte-long lines, unclosed tags and entities of every shape. Three review rounds on this parser established and tested the following, each pinned by a test over the sources:

  • No regex crosses a line without a bound. Two passes that once scanned to the end of the document on [\s\S]*? were replaced by find() scans after 44 KB of period-free boilerplate overflowed the C stack; a hygiene test now asserts no unbounded cross-line regex exists in the SIU sources.
  • Every pass is linear. Two tag-stripping passes were quadratic (re-scanning from each unclosed <script or bare <); at the 2 MiB input cap one took 40 minutes. Both are single-pass now.
  • Lines are capped at 2,000 characters and split at the last sentence end inside the window, else the last space; the functions warn how many lines were split, because a field spanning a split can come back incomplete.
  • Numbers are bounded. Every string-to-integer conversion goes through one helper that reads up to nine digits; SO #1234567 is 1234567, and SO #4444444444444444 is ignored as a tag (the count falls back to the other cues), not an overflow.
  • The user can interrupt. The whole path polls for an interrupt through a hook the host installs, and raises it only from a frame that owns no live C++ objects.
  • Input is capped at 2 MiB, and the fuzzers run the parser under an 8 MB stack like R’s, with seeds for every shape the reviewers used.

Fetching live reports

siufetch() fetches a report by its identifier through the package’s checked transport (address validation, redirect policy, resource limits) with an Internet Archive fallback, and siufprs() fetches and parses in one call:

f <- siufprs("24-OCI-123")

The corpus of parsed reports, reviewed round by round, is distributed as a dataset by the companion package rmoriedata; this package provides the parser so the corpus can be rebuilt and audited.

Indigenous police services

Ontario’s First Nations police services, among them the Nishnawbe-Aski Police Service, the Anishinabek Police Service and the Treaty Three Police Service, are investigated by the SIU like any other. The parsed police_service field selects their cases (this fetches live reports, so it is not run here):

fnservices <- c("Nishnawbe-Aski Police Service", "Anishinabek Police Service",
                 "Treaty Three Police Service", "UCCM Anishnaabe Police Service")
reports <- lapply(reporturls, siufprs)
mine <- Filter(function(r) any(grepl(paste(fnservices, collapse = "|"), r$police_service)),
               reports)
length(mine)

The report’s timeline in a picture

siuprs() returns each date in the report in ISO form. Drawn on one axis, they give the investigation’s timeline at a glance.

rpt <- siuprs(siutxt(paste(readLines(system.file("extdata", "siu_synthetic_report.html",
                                                    package = "rmoriebricklayer"), warn = FALSE),
                           collapse = "\n")))
iso <- rpt[grepl("_iso$", names(rpt)) & grepl("^\\d{4}-\\d{2}-\\d{2}$", rpt)]
when <- as.Date(iso)
o <- order(when)
lab <- gsub("_", " ", sub("_iso$", "", names(iso)[o]))
span <- diff(range(when))
op <- par(mar = c(4, 1, 3, 1))
plot(when[o], seq_along(o), pch = 19, col = blpal()[1], yaxt = "n", xaxt = "n", xlab = "", ylab = "",
     xlim = range(when) + c(-0.1, 0.1) * as.numeric(span), ylim = c(0.5, length(o) + 0.5),
     main = "Dates in the synthetic report")
axis.Date(1, when, format = "%Y-%m-%d")
text(when[o], seq_along(o), paste0(lab, "  ", format(when[o])), cex = 0.75,
     pos = ifelse(seq_along(o) < length(o), 4, 2))

par(op)