The narrative corpus behind morie_describe() was excluded from the
0.3.1 tarball to get this package under 5 MB, which left rmorie
carrying the only shipped copy. rmorie has no room for it either, so
the corpus is back where it belongs, and both packages sit in the
5-10 MB range CRAN accepts with a written justification.
The package shipped 9.2 MB, and most of the excess was duplication:
rmoriedata.sqlite (15 MB uncompressed) was retired by the
SQLite-to-Parquet migration and no shipped code reads it. Kept in the
repository, excluded from the tarball..csv.gz. load_siu_reports(format = "parquet") now reads the store
copy that morie_data_load() already serves, so the top-level
duplicate is gone. The CSV remains: it is the default format and a
genuinely different encoding, not a third copy of the same one.describe_corpus.Rds (1.6 MB, and it does not compress) is referenced
by no R code, Rd page or test. Excluded.Verified after the change: the Parquet path, the CSV path and the
siu_directors_reports store slug all return the same 5,157 x 65 corpus.
nanoparquet is gone from Imports. The bundled store is now read and
written by this package's own codec (R/aaa_parquet.R), so a plain
install no longer pulls a compiled Parquet dependency, and
load_siu_reports(format = "parquet") works on a machine that never
had one -- previously it turned a missing optional package into a hard
stop, despite the corpus being bundled in exactly that format.morie_data_load("vic_<key>"); they appear
in morie_data_catalog() like any other slug.data-raw/build_vic_tables.R.morie_datasets_vic_table() defaults to offline = TRUE, and with
nothing bundled it returned a 0-row frame on any machine that had not
already downloaded the workbook.hostload() returns a table as its file holds it. It read the file with read.csv()'s defaults, so codes lost their leading zeros (046 became the number 46), column names were rewritten (Offence Description became Offence.Description) and the text NA became a missing value. Now the names are kept as written, an empty field is missing in a logical or number column and "" in a text column, and a column becomes logical, integer or double only when every value in it is written the way R writes such a value; any other column stays text. Text is marked UTF-8 in every locale (it was unmarked in a non-UTF-8 locale), and bytes that are not UTF-8 are marked "bytes". The help page says exactly which values count as numbers.hostload() failed with a network error.hostload() checks a cached table under the same lock as a download, so it can no longer delete a copy that another R session has just refreshed. A session waiting for another session's download stops with an error once that download has not grown for options(rmoriedata.lockwait = ) seconds (600 by default); it waited for ever behind a stalled session.hostcat() or hostload()), the table and the address, without the dependency's own function name in front (bldl:).saveRDS() and read back, passed through serialize(), or rebuilt from its fields) is refused like a budget from another session: the copy and the original could each spend the whole budget. dpbudget()'s help says what is and is not prevented.parquet: error naming the column: it marked such bytes as UTF-8, and nchar() or toupper() failed later. columns = character(0) gives the file's rows with no columns (it failed with "footer says 3 rows, decoded 0"), and a damaged footer length is reported as damage, not as a file over the row cap.chiload(full = TRUE) reads text as decimal numbers only, so "0x1A" is no longer 26 and "Inf" or "1e400" no longer an infinite coordinate. A value in Socrata's export that does not fit its column becomes NA with one warning per column, instead of refusing the whole download; a mirror's or the cache's file in the sample's columns must still fit exactly, and the error names that file (it blamed Socrata's export).mdask() no longer probes or asks an Ollama server the user did not choose: with the automatic route, a local server at localhost:11434 is left out unless route = "ollama", an OLLAMA_* variable or a saved Ollama setting names it.cihiget() checks timeout under its own name and takes 1 to 86400 seconds (0.5 was refused as "tmout must be a positive number"), and a failed download names cihiget(), not the dependency's blfetch(); strings marked "bytes" passed to kanonvrf(), ldivvrf(), mdload(), mdpath(), mddict(), cihiget(), hostload() and mdask() are refused in words, where base R stopped with "translating strings with "bytes" encoding is not allowed".mdload(), hostload() and chiload(full = TRUE): R's own conversion (behind read.csv() and as.numeric()) is not correctly rounded, and read some values one bit apart on x86-64 and ARM64 (1.239042694 was 1.2390426940000001 on one and 1.2390426939999999 on the other). The Parquet copies of five bundled tables (tps_assault_sample, samples_tps_assault_sample, vic_criminal_incidents, vic_lga_criminal_incidents, vic_recorded_offences) held such values and are written again from their CSV copies, so the two copies of every table give the same data frame, as the help says.kanonvrf() and ldivvrf() compare every numeric column exactly, whatever its class: difftime, I(), Date, POSIXct, POSIXlt and complex columns were compared to 15 significant digits, so two distinct values (such as two times a microsecond apart) formed one class and a violated k could be reported as satisfied. A fractional Date is now compared by its stored value, not its printed day. Columns that cannot be compared exactly (bit64::integer64, list and matrix columns) are refused with an error naming them.kanonvrf() and ldivvrf() build classes column by column, so strings holding control characters can no longer join two different rows into one class, and a factor level that is NA is missing while the string "NA" is a value (they were one class). As a sensitive value, an NA factor level is withheld and no longer counts towards diversity.delta = 0 refuses every release with delta > 0: a release at delta = 5e-13 was accepted. The check of the totals allows only floating-point rounding, relative to the budget, and never a release more than twice what remains: a budget of epsilon = 1e-13 paid for a release at 1e-12.Rscript started as a container's entrypoint does: each budget carries a random token drawn from the operating system when the package loads. Budgets saved by earlier versions are refused too. dpbudget()'s help says that a release interrupted after it is charged stays charged.chiload(full = TRUE) reads a text time with an offset or Z (from a mirror) as that instant: 2024-06-01T10:00:00Z was read as 10:00 in Chicago, five hours off. A value that does not fit its column's type is refused with an error naming the column, from a mirror and from Socrata's export alike: text "1.5" became the integer 1, 2 became TRUE, and junk in Socrata's numeric columns became NA..parts folder and could delete or overwrite each other's parts, and an unchecked table sat under its final name until its checksum was read. A download that is cut short or killed still resumes from the parts it finished. The manifest is written under a name of its own and renamed, so sessions fetching it at once no longer write into one file.integer64 columns, naming them: a matrix column was written with more values than rows, which other readers cut short. Dates and times are written down to the whole day and millisecond as R prints them, so a Date of -0.5 (printed 1969-12-31) no longer reads back as 1970-01-01, and dates or times outside what Parquet holds, or infinite, are refused.compression = "zstd" by nanoparquet, arrow or DuckDB, with no other package: the Zstandard decoder is in the package, written from RFC 8878 (no dictionaries), and checks every page against its stated size, and against the frame's checksum when it has one. Such a file stopped with "compression codec 6 not implemented". The writer still writes GZIP, Snappy or no compression.options(rmoriedata.cache_dir = ), where it was a short file that then failed its checksum. Each part is deleted once it is joined, so the download needs the table's size plus one part on disk, not twice the table's size.chiload(full = TRUE) returns the bundled sample's columns, in its order and with its types (16 columns for complaints and 8 for arrests, with date_iso and a POSIXct date), derived from Socrata's export the way the samples were: it returned Socrata's raw 22 or 24 columns. Socrata's export is read as text first, so an IUCR code keeps its leading zero.chiload(mirror = ) with a URL downloads <type>_full.parquet from the mirror over the package's checked https transport, and checks it against <type>_full.parquet.sha256 when the mirror has one; a mirror that fails or does not match falls back to Socrata. A URL was taken for a missing local file, so the mirror was never used.chiload() checks its arguments in words: limit must be a whole number from 1 to 2147483647 (Inf and 1e10 failed with a coercion error), fraction a number above 0 and at most 1, and full and refresh TRUE or FALSE (full = "yes" or NA silently returned the sample).cellsupp() checks which hidden cells the totals determine with one factorisation for all of them, not one per cell: a 50x50 table took 152 seconds and a 60x60 table did not finish in 17 minutes; both now take well under a second, with the same cells suppressed. Negative counts are refused, and the help says that zero cells are published as they are.mdload(slug, refresh = TRUE) checks the file against the signed manifest again: a bundled file changed after its first load in a session was still returned.hosted/manifest.json that does not parse is fetched again, once, with a message; it blocked hostcat(), hostload() and hostload(refresh = TRUE) until it was a day old.hostload() takes a large table whole when the server answers a byte-range request with the whole file, instead of failing on every run, and a failed run leaves no empty .parts directory.dplaphist() keeps the names and dimensions of its counts, so a table()'s labels stay on the release.parquet: message: about 4% of corrupt files stopped with a base-R message such as "subscript out of bounds", and a footer of 5000 nested lists overflowed the stack. Its help, and chiload()'s, say that a POSIXct column reads back in UTC (the same instant): Parquet keeps no time zone.k or l of Inf or above the integer range in kanonvrf() and ldivvrf(), an epsilon above 2^20 in the counting mechanisms (both stopped with "missing value where TRUE/FALSE needed"), a slug that is not one name in mddict(). mdsha() takes one string (it hashed only the first of several) and hashes it as UTF-8 whatever encoding it is marked with.dplapcnt(), dplaphist() and dpgmean() draw fresh random bytes in a forked worker (parallel::mclapply()): the workers inherited the parent's buffer of random bytes and released identical noise, so the difference of two releases was the exact true difference.chiclr() deletes only what chiload() wrote (the <type>_full.parquet files, their checksum files and unfinished writes): it removed everything in the cache directory, subfolders and the hosted-table cache included.cellsupp() refuses a table of three or more dimensions (it was flattened to one column and suppressed against the wrong margins). ldivvrf() counts a class's values in one pass: 200,000 classes took minutes.hostlogin() is gone, and no key is read or sent: every table at data.rmorie.com is open, so the sign-in asked for a name, a reason and an email or GitHub account for nothing. hostcat() and hostload() work as before. A key that an earlier version stored stays in its file, unread. The manifest and services fields are read by exact name.hostload() checks every table against the manifest's SHA-256 again, whatever the field is called (sha256_gz, sha256 or sha, as rmoriebricklayer reads it): 0.4.2 read sha256 only, so 631 tables listed with sha256_gz were used unchecked and a modified cached copy loaded. A digest field that is not a SHA-256 is an error, and a table listed without any digest loads with a warning, once per session.hostcat(), with a message counting them, and hostload() refuses them before any request. The A2AJ case law is an ordinary open table.dplapcnt(), dplaphist(), dpgmean() and dpbudget() refuse an infinite epsilon in words; dplapcnt(5, Inf) stopped with "missing value where TRUE/FALSE needed".~/.config/R/rmoriebricklayer.dpbudget() cannot be spent from a forked worker or another R process: dplapcnt(), dplaphist() and dpgmean() refuse it there with an error. Four workers under parallel::mclapply() each spent 0.9 from one budget of 1 and the parent recorded nothing; give each worker its own budget and add up the epsilons.kanonvrf() and ldivvrf() compare numeric quasi-identifiers exactly: values that agree to 15 significant digits, such as 0.1 + 0.2 and 0.3, were one class, so a class of one could be reported as a class of five. -0 and 0 are one class, and NaN is missing with NA.cellsupp() treats an NA cell as unpublished in its audit, as its help says, and gives it complements like any hidden cell: it was treated as known, so a hidden cell recoverable through it was reported as protected. Infinite and fractional counts are refused, and a function or an environment gets the package's "must be a numeric matrix or table" error.chiclr() deletes only arrests_full.parquet, complaints_full.parquet, unfinished write-*.parquet files and their .sha256 files: it deleted any file ending in _full.parquet and left the .sha256 of an unfinished write.chiload(full = TRUE, mirror = ) gives a mirror's table the bundled sample's types when its columns have the sample's names (a date stored as text came back as text), and refuses a column that cannot be read as the sample's type.hostload() and mdload() check refresh in words (NA, "yes", NULL and c(TRUE, FALSE) gave base-R errors or acted as TRUE), and a table size in the hosted manifest that is not a whole number from 1 to 2^40 bytes is refused in words: an absurd size stopped in seq().mdverify() and mdsums() list hidden files in the store too, and the help of mdverify() says what it covers: every file of the data store, not the package code or the lazy-loaded complaint_sample and arrest_sample. mdsums(), siuload() and cihitbls() read the store named by options(rmoriedata.store = ), the one they verify; they used to read the installed copy.dplapcnt() and dplaphist() refuse a count of 2^53 itself: 2^53 + 1 rounds back to 2^53, so the release could equal the truth.NaN (it read back as NA), refuses a data frame with no columns, which its reader could not open, and refuses list, complex, raw and difftime columns in words, naming the column: they were written as text. Every error of the reader starts with parquet:, and a damaged file no longer lets a base-R warning such as "NAs introduced by coercion to integer range" escape before the error.pyreadr and the right hostcat() columns, and drops an out-of-date note on package size.yoy() is renamed yoychg()): rmoriedata no longer refers to yoy().hostload() downloads a table larger than twice the part size (option rmoriedata.dpart, 90 MB by default) in byte-range parts, each kept once whole, so a broken connection costs at most one part and running it again resumes; the joined file is checked against the manifest SHA-256.hostcat() and hostload() need no sign-in; a key stored earlier is still sent. ICPSR studies are no longer served by data.rmorie.com (ICPSR's terms do not allow redistribution); asking for one says where to get it.hostlogin()'s help names the arguments rmoriebricklayer::dlogin() takes (affil, purp, restr; it listed affiliation, purpose, restricted). A table refused with 403 is reported as a key asked for or refused: no table sits behind accepted terms any more.R CMD check on R before 4.6 no longer notes rmoriebricklayer as an unused import.morie_data_load() is mdload(), morie_dp_laplace_count() is
dplapcnt(), load_chicago_data() is chiload(), quasi_identifiers is qids, and so on.
Each help page gives the full form of its short name. The old names are not kept.hostlogin()), mdask() and the downloads that carry the data key need
rmoriebricklayer 1.0.0; with an older one they say so, and attaching rmoriedata prints a
one-line note on how to update.morie_data_hosted_login() needs the data
access terms accepted first, then your name, organisation and why you need the data (and, for the
restricted tables, a supervisor, department chair, director or coordinator). Pass them as
arguments (accept_terms = TRUE, name = , purpose = , ...) or answer at the console.ask() is built on (0.5.9 was
accepted, so a build against an older one ran ask() on the retired hosted tier).ask() uses a language model you bring (a provider's key, Ollama, or your own server) through
rmoriebricklayer, and the model is given rmoriedata's and rmoriebricklayer's help with tools to
read it. Its backend argument is now route ("auto", "own", "ollama" or a provider id
such as "anthropic"); the hosted MORIE tier and the rmorie command-line fallback are gone.morie_data_hosted_load() uses a downloaded table only when it matches the SHA-256 the
manifest lists (fetching the manifest again once before calling a file bad); a mismatch
deletes the copy and says so. A restricted table (the ICPSR studies, NCVS 38604, the a2aj
case law) refused with 403 says to accept the terms at https://data.rmorie.com/terms.morie_data_llm_config() shows and saves the language-model settings (route, and the
address, key and model of each route), shared with rmoriebricklayer, rmorie and morie.A hex logo (man/figures/logo.png, source data-raw/hex_logo.svg), on the README and the
package website, with its favicons; the website indexes morie_dp_budget() and morie_dp_spent().
A garbled download is refused, not padded. The full Chicago export and the tables from
data.rmorie.com are read with fill = FALSE: a row with a field missing or one too many was
padded into the result (and, for Chicago, written to the cross-session cache); it is now an
error, and a damaged hosted table's local copy is removed so the next call fetches it again.
The Chicago tests also pin the 1% row-count tolerance at its edge (495 of 500 accepted, 494
refused) and the request size (the service's count, at least 5,000,000 rows).
Privacy verifiers (review of 0.3.5). morie_k_anonymity_verify() treats a missing
quasi-identifier as a level of its own: stats::aggregate() had dropped every row with an NA
key, so a class of one person could pass k = 3; class sizes now always sum to nrow(data).
morie_l_diversity_verify() counts only known sensitive values ("HIV", NA, NA is diversity 1,
not 2). morie_cell_suppress() adds complementary suppressions to a fixed point and then audits
the result exactly: a hidden cell the row and column totals determine (its indicator in the row
space of the margin constraints) gets another complement, and the result carries
recoverable_mask, n_recoverable and a protected verdict to test before publishing. The
0.3.5 passes left a lone hidden cell in a row or column, from which the primary value followed
by two subtractions; zero cells are no longer used as complements. Checked against brute-force
enumeration of every margin-consistent fill on 464 random tables: no disagreement.
Differential privacy. Noise comes from the operating system's CSPRNG
(rmoriebricklayer::random_bytes()), not R's seeded generator, so set.seed() no longer
reproduces a release. Counts and histograms use the exact discrete Laplace sampler of Canonne,
Kamath and Steinke (2020) in integer arithmetic (whole-number releases; no floating-point
artefact of the kind Mironov (2012) showed in the textbook Laplace sampler). The mean uses the
analytic Gaussian mechanism of Balle and Wang (2018), valid for every epsilon; the classical
bound, valid only for epsilon <= 1, is gone. All three mechanisms use one neighbouring relation
(add or remove one record): the mean is a noised sum over a noised count. New
morie_dp_budget() / morie_dp_spent() charge releases against a budget (basic composition)
and refuse one that would exceed it. Examples no longer call set.seed().
Parquet reader. Every byte access is bounds-checked (an out-of-range raw index had silently
read zeros); varints, lists, maps and nesting are capped by the bytes that remain, so the 21-byte
file that looped forever is refused at once; the decoded row count must equal the row group's
and the footer's exactly (no padding with NA, no truncation); a column chunk's type must match
the schema's; PLAIN values must fill their page exactly; dictionary indices are range-checked;
page sizes and footer row counts are capped (options(rmoriedata.parquet_max_page_bytes, rmoriedata.parquet_max_rows)); PageHeader.crc is verified when present and the writer now
writes it. tests/testthat/test-parquet-hostile.R fuzzes all three codecs.
Downloads. .rmd_dl(), which carries the MORIE key as a bearer header, goes through
rmoriebricklayer::bricklayer_download() (https to public hosts only, no file://, size cap)
instead of base url(); requires rmoriebricklayer 0.5.9. load_cihi_data_tables() checks the
catalogue against the signed manifest before reading it. morie_data_verify() reports files
added to the store (listed = FALSE), and morie_data_checksums() compares each digest with the
manifest (in_manifest).
Workflows. The SIU manifest refresh runs only on demand and uploads an artifact instead of
committing: the manifest is covered by the XMSS-signed _checksums.csv, whose key never leaves
the maintainer's machine, so it is re-signed locally with data-raw/sign_store.R. A new
ASAN/UBSAN job builds the C and runs the suite, including test-codec-hostile.R.
Packaging. utils is declared in Imports; the stale RoxygenNote is dropped
(Config/roxygen2/version is the roxygen 8 field); NOTICE moves to inst/NOTICE so it
installs, and its data attribution now names the publishers instead of per-file headers that did
not exist. The licence text itself stays out of the tarball, as CRAN asks for a standard licence.
DESCRIPTION describes the disclosure-control helpers by what they do.
The curated-data service address comes from the signed services document at
rmorie.com (through rmoriebricklayer::bricklayer_services() when the installed
bricklayer has it; an older bricklayer keeps the default address), so the endpoint can
move or be paused without a release. Keys are personal and issued on request at
www.rmorie.com/access/; every hint says so, and morie_data_hosted_login(token = )
stores one (the GitHub and emailed-code sign-ins keep working).
ask() follows rmoriebricklayer::bricklayer_llm_ask()'s route order: an endpoint of
your own, a local Ollama server, then the hosted MORIE tier as a last resort;
backend = "hosted" insists on the hosted tier. The setup hint when nothing answers
names all three.
The Parquet store is compressed: the writer's Snappy stream was literal-only (valid, but it
compressed nothing), so every table's Parquet copy was as large as its raw pages. Pages are now
written with GZIP (zlib, level 9) by default, and Snappy, still available, has a real
match-finder; both codecs and the reader are C (src/gzip.c, src/snappy.c). The Parquet copies
shrink from 16.1 MB to 3.3 MB (the SIU corpus from 10.0 MB to 2.1 MB) and the installed package
from about 27 MB to 14.5 MB; pandas/pyarrow read them as before.
SIU corpus: narrative_summary holds each report's own account of the incident on 4,605 of 4,613
reports (it was empty but for three values that described the SIU's investigation). The text is
the opening of the report's "Incident Narrative" / "Description de l'incident" section, or of the
"Notification of the SIU" / "Avis à l'UES" section in reports that have no narrative section (2005-2016
and re-opened files), about 900 characters cut at a sentence end. It was read from the SIU's pages,
re-crawled on 2026-10-05 at one request every 3 seconds; data-raw/add_siu_narratives.R rebuilds it.
SIU corpus, round 8 (4,613 rows; every change is a row of data-raw/siu_round8_review.csv, with
the reason read from the page, or a rule of data-raw/siu_round8_rules.R):
corrigenda, not counted as
a report (drids 385, 620/621 and 5065 correct 328, 521/522 and 4561); six second copies of one
report (identical page text) are removed;charges_recommended is "FALSE", "TRUE" or empty (it had 39 spellings), and logical in the
typed table (morie_data_load("siu_directors_reports"), CSV and Parquet), as panel_reviewed is;
sex_gender_affected is male or female; one name for Kawartha Lakes and for Cornwall (the
Cornwall Community Police Service is now the Cornwall Police Service);narrative_summary held the SIU's mandate paragraph or the page title, not a narrative, and is
empty except on the three rows that held report text; supplemental_materials lists the
legislation and case law a report cites (it held the page's share and privacy links).ask() sends the bundled catalogue (slug, kind, rows) with the question, so the model names tables
(it was told only to "prefer the bundled catalog", which it could not see). Through the rmorie binary it
calls rmorie agent [--model NAME] QUESTION, the agent's own options: it passed --backend, which
the agent refuses, and -m for the model; backend is "auto", "hosted" or "cli". The examples that need a stored key or the binary are \dontrun; the one that runs
switches the hosted tier off (MORIE_HOSTED_BASE_URL = "off"), so it reaches no network
whatever key the machine holds.
morie_dp_laplace_count() and morie_dp_laplace_histogram() refuse Inf in words (they warned
about integer coercion, then failed in R's internals) and refuse counts above 2^53: a double that
large has no room for Laplace noise of scale 1/epsilon, so morie_dp_laplace_count(1e300, 1)
returned exactly 1e300, a "private" release equal to the true count. A whole count beyond R's
integer range but below 2^53 is still a count. morie_core_mean() refuses text instead of
averaging NA.
Help: ?rmoriedata describes the package's functions (it said there were none); ?load_siu_reports
lost "% of them" to an Rd escape; "more than 160" databases at data.rmorie.com; a garbled sentence
in ?load_chicago_data; the README's Chicago example is bounded (limit = 200).
The SIU director's-report corpus: police_service is the service of the subject officials
on every row (it was the notifying force on many French rows, and on 24 English rows a
custody, requesting or neighbouring service -- each corrected after reading the report);
one spelling per service; 59 English reports published since the last build, reviewed by
the same model panel, and their 60 French reports added; 54 French reports filed under
/en/ relabelled; 29 rows that held no director's report removed.
ask() answers through the hosted MORIE tier with the key
morie_data_hosted_login() stores (ollama.com cloud models and additional AI
models such as kimi-k2.6:cf or gpt-oss-120b:cf), and only falls back to the
optional rmorie command-line agent when no key is stored.
Hosted tables download with a live progress bar (percent, size, rate) in
an interactive session; options(morie.quiet = TRUE) silences it.
Requires rmoriebricklayer 0.5.2 or newer (the login flow).
morie_data_hosted_login() signs in to the hosted MORIE tier (GitHub device
flow, email code, or a key you hold) and stores the key in the credentials
file every MORIE package reads, so rmoriedata alone is enough to use
data.rmorie.com.
morie_data_hosted_catalog() and morie_data_hosted_load("db/table"): the
160 databases the MORIE project materialises from BigQuery public datasets,
served from the edge at data.rmorie.com and opened by the MORIE key that
rmorie, morie or rmoriebricklayer store. Cached under the package cache
directory (tempdir() unless options(rmoriedata.cache_dir = )).
Nothing is written outside tempdir() unless the user asks. The
Chicago full-dataset cache lives under tempdir() by default and moves
to a persistent directory only through
options(rmoriedata.cache_dir = ); clear_chicago_cache() empties it.
load_chicago_data(as = "parquet_path") also writes under tempdir():
it wrote the bundled sample into the user's R_user_dir() cache, and a
bounded full = TRUE fetch could land on the full-dataset cache path.
The schema names the language column of siu_directors_reports and
siu_drid_manifest _language, as the files do; it carried the
X_language that check.names had made of it, so the typed store and
load_siu_reports() disagreed on the name. load_siu_reports() documents
that it returns the corpus as text ("" for empty cells) while
morie_data_load("siu_directors_reports") returns the typed frame.
Every bundled table ships as Parquet again, next to its CSV
(inst/extdata/parquet/<slug>.parquet, plus _catalog.parquet and
siu_directors_reports_corpus.parquet). morie_data_load(format = "parquet") and
load_siu_reports(format = "parquet") read those copies; the new
morie_data_path(slug, format) returns the verified path of either
copy, which is the bridge to pandas.read_parquet(). The catalog gains
a parquet_path column and the signed manifest covers the new files.
The Parquet writer no longer mangles non-ASCII text in a C locale
(enc2utf8() on an unmarked string turned each byte into its <c3><a9>
display form, and load_chicago_data(full = TRUE) wrote that into the
cross-session cache). load_siu_reports() reads its CSV as UTF-8 like
the store does, so both copies agree in every locale; a data page v2
file is reported as such before any decompression; and CI runs the
check in a C locale. Every CSV reader in the package now declares
UTF-8, so a live Chicago fetch and its cached copy are identical()
in any locale. A string that is not valid UTF-8 and carries no
encoding mark is refused by column name (a Parquet string is UTF-8;
escaping lost the data, passing the bytes through made a file no other
reader opens), and duplicate column names are refused on write and kept
apart on read. The cross-session Chicago cache carries a SHA-256
sidecar checked on every read. load_cihi_data_tables() rejects an
archived_only that is not TRUE or FALSE instead of ignoring it.
morie_data_checksums() gains a path column relative to the extdata
root; the bare file basenames it returned were not unique and 20 of
them did not resolve from the root (morie_data_verify() already
returned path).
ask() quotes every argument it hands to the rmorie CLI. The
preamble contains a parenthesis, so with the CLI installed every call
had died in the shell before the binary ran; a ; in the question
would have run as a command. model and backend are validated, and a
blank or NA question is rejected like "" and NULL. A stub binary
on PATH now exercises the branch the tests never reached.
morie_data_load() on a dictionary slug says to use
morie_data_dictionary() instead of pointing back at the catalog.
load_chicago_data(full = TRUE) no longer caches a response that is not
the dataset. A 200 carrying an outage page, a header-only body or an
export shorter than the row count the service reports is refused with
the reason; before, an outage was written to the cross-session cache and
every later session read the empty frame back. The cache is written to
a temporary file and renamed, so a concurrent reader never sees a
half-written file; a cache that cannot be read is discarded and
refetched; and a new refresh = TRUE argument refetches on demand. When
the service's count endpoint is down the data is returned but not
cached across sessions.
fetch_cihi_table() checks the downloaded file's size and signature
against the catalogue's declared format and validates which (a whole
number within the catalogue, or a non-empty title substring) and
timeout before any request. A live URL that answers 200 with an
outage page now falls through to the Wayback copy.
The verifier pins the signing key's XMSS root and public seed in package code. A store re-signed with another key, even with a consistent manifest and shipped public key, no longer verifies; the signature now means more than a checksum against local tampering.
CSV tables are read with encoding = "UTF-8" instead of
fileEncoding = "UTF-8-BOM": in a C locale the latter truncated 22 of
the 99 tables (5 to zero rows); the bytes are now left alone and
marked as UTF-8 in every locale.
The data store is signed. _checksums.csv lists every shipped file
with its SHA-256, and the manifest's own SHA-256 carries an XMSS
(RFC 8391, SHA-256) signature whose public key ships as
_signing_key.json. The signature is checked once per session and each
file when first read, so a modified or corrupted install errors instead
of returning altered data; morie_data_verify() checks the whole store
at once. Needs rmoriebricklayer 0.4.0 or newer for the verification.
load_siu_reports(format = "parquet") pointed at the parquet copy this
release removed; the option now errors with a note that the gzip CSV is
the same corpus. rmorie reads the corpus through the default "csv".
morie_dp_gaussian_mean() now rejects infinite lower or upper with
the error its message already promised, instead of returning NaN with
a warning from rnorm().
The data store ships every table once. morie_data_load() reads the
table's CSV (the file rmorie reads by name) and applies a bundled column
schema, so each table comes back with exactly the names and classes the
retired parquet copies had (verified identical for all 99 tables), and
keeps it in a session cache: repeated loads are instant. The parquet
directory, a duplicate samples/ directory and a second copy of the SIU
reports are gone, which takes the source tarball from 6.8 MB to under the
5 MB CRAN cap. The ten Victoria tables that existed only as parquet now
ship as CSV under vic/. morie_data_catalog() has n_rows and
n_cols for every table; morie_data_dictionary() returns the shipped
JSON.
Documentation is generated with markdown roxygen; backticks in the help pages are now \code{} (they rendered as stray opening quotes in the PDF manual).
load_chicago_data(full = TRUE) gains limit (exact row cap) and
fraction (share of the live dataset, total looked up first); bounded
fetches never touch the full-dataset cache. Network examples are now
\donttest{} with try() so they degrade gracefully offline.
SIU corpus also bundled as Parquet; load_siu_reports(format = "parquet") reads it via nanoparquet (columnar / SQL-friendly). CSV remains the default (no extra dependency).
panel_reviewed flag column (65 cols).inst/extdata/rmoriedata.sqlite
(retired by the sqlite-to-parquet migration; no shipped code read it —
the loader API is Parquet-only via nanoparquet, and the file remains
backed up outside the package). Tarball drops from 5.5 MB to 4.25 MB.AGPL (>= 3) (the LICENSE file was a
verbatim copy of the standard license, which CRAN flags).CITATION.cff, NOTICE, and LICENSE are excluded from the built
package (.Rbuildignore), clearing the top-level-files NOTE.LinkingTo: rmoriebricklayer (>= 0.2.0) and links
the ecosystem's shared compiled core instead of duplicating any C code.morie_core_sha256() and morie_core_mean() call the shared
kernels directly (fast data-integrity hashing + summaries for the integrated
fixtures, with no dependency on rmorie). Tests assert they are
byte-identical to rmoriebricklayer's own core_sha256() / core_mean().Six small, base-R-only helpers for analysts releasing aggregate statistics from the integrated fixtures (or any other dataset) without re-identification risk:
morie_dp_laplace_count() — (epsilon, 0)-DP count via the Laplace
mechanism (sensitivity 1).morie_dp_gaussian_mean() — approximate (epsilon, delta)-DP mean of a
bounded numeric vector via the analytic Gaussian mechanism.morie_dp_laplace_histogram() — per-bin Laplace noise on a histogram.morie_k_anonymity_verify() — checks k-anonymity over a set of
quasi-identifiers and returns the offending equivalence classes.morie_l_diversity_verify() — per-class distinct-count check on a
sensitive attribute.morie_cell_suppress() — small-cell suppression with optional
complementary suppression so primary-suppressed cells can't be
reconstructed from row/column marginals (StatCan-style).All helpers use only base R + the existing stats import; no new
runtime dependencies. Inputs are validated and edge cases (empty
vectors, NAs, out-of-bounds inputs, invalid privacy parameters) throw
informative errors.
tests/testthat/test-dp.R — Monte-Carlo convergence + variance-scaling
sanity tests for the DP mechanisms; input-validation coverage.tests/testthat/test-k-anonymity.R — hand-built fixtures with known
equivalence-class structure; complementary-suppression correctness.