
From a naive LLM judge to a rigorous one
methodology.Rmd
library(liminal2)
library(arrow)
#>
#> Attaching package: 'arrow'
#> The following object is masked from 'package:utils':
#>
#> timestamp
library(dplyr)
#>
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#>
#> filter, lag
#> The following objects are masked from 'package:base':
#>
#> intersect, setdiff, setequal, union
library(tidyr)Purpose
This vignette explains how the scoring method evolved and what the changes did to the measurements. It is written for users who want to understand why the package does not simply ask a model “how big is the liminal space?” and trust the answer.
The story has four stages:
- a naive single-shot judge, which is the obvious design and the one we deliberately did not ship;
-
version
1.0.0, which decomposed the score into seven anchored, independently scored dimensions; -
version
1.1.0, which added a scope guard and an ordered scoring procedure to every dimension prompt; -
version
1.2.0, which removed the hard floors and cross-dimension double counting that1.1.0introduced, added a structural calibration rule, and sharpened the procedures that had conflated method uncertainty with decision disagreement.
All three prompt versions were benchmarked over the same six
scenarios, the same three models and the same two repetitions. The
1.0.0 and 1.1.0 caches are frozen in
inst/extdata/model-comparison-results-v1.0.0.parquet and
inst/extdata/model-comparison-results-v1.1.0.parquet; the
current cache is
inst/extdata/model-comparison-results.parquet. All three
ship with the package, so this vignette renders offline and the
comparison is reproducible.
Stage 0: the naive judge
The naive design is one call to a language model with a prompt like:
“How large is the liminal space between this business ask and a data-based solution? Answer from 0 to 10.”
It is cheap to write and it fails in five distinct ways.
| Failure mode | What goes wrong |
|---|---|
| No decomposition | Seven different obstacles collapse into one number. A large score cannot say whether the ask is ambiguous, the data are unsuitable, or the decision rule is missing, so it cannot guide any action. |
| No fixed scale | Nothing defines what 0 and 10 mean, whether 0 is good or bad, or how this case compares with the next one. Two runs on the same ask can land on either side of the midpoint. |
| No reference class | The model has no common yardstick across business cases. Scores drift with prompt wording, model version and the order of the conversation. |
| Free-text output | The answer arrives as prose (“around a 6 or 7, depending on the data”), so it must be parsed heuristically. Parse failures and silent coercions become measurement error. |
| No provenance | The result records neither the model, the prompt text, nor the scale, so two scores cannot be known to be comparable. |
None of these are model failures. They are measurement-design failures: the naive judge produces an opinion, not a measurement. We did not benchmark this stage, because there is nothing stable to benchmark.
Stage 1 (1.0.0): decompose, anchor, constrain
Version 1.0.0 turned the opinion into a measurement with
six design decisions:
-
One prompt per dimension. Seven independent calls,
each scoring exactly one of the dimensions in
liminal_dimensions(). A failure or ambiguity in one dimension cannot contaminate the others. - A fixed direction statement. Every prompt states, in identical words, that 0 is the smallest liminal space and 10 the largest, and that the answer must be an integer.
- Fixed anchor cases. Every dimension has three generic anchors at scores 2, 5 and 8. They describe structurally distinct situations, not named projects, and are versioned with the prompt.
-
Structured output. The wire schema is a single
integer field,
score, so there is no free-text parsing. - Determinism controls. Temperature 0, a fixed default model and stateless calls. (The gateway is not bitwise deterministic even at temperature 0; the tests bound the jitter rather than demanding identical repeats.)
- Provenance and anti-circularity. The result records the model, base URL and prompt versions. The six validation scenarios are used only to check the ordering of the final measure, never to write or tune prompts.
The aggregation is the geometric mean implied by the product rule:
where is the condition score of dimension and its weight.
1.0.0 was benchmarked with 252 calls (three models, six
scenarios, seven dimensions, two repetitions). Every call returned a
valid score; no retry was needed. But the benchmark also exposed a
systematic problem.
What 1.0.0 still got wrong
The scenarios are designed so that liminality rises from S1 to S6. In
the 1.0.0 run it did not:
read_cache <- function(file, prompt_version) {
arrow::read_parquet(
system.file("extdata", file, package = "liminal2")
) |>
mutate(version = .env$prompt_version)
}
results <- dplyr::bind_rows(
read_cache("model-comparison-results-v1.0.0.parquet", "1.0.0"),
read_cache("model-comparison-results-v1.1.0.parquet", "1.1.0"),
read_cache("model-comparison-results.parquet", "1.2.0")
) |>
mutate(scenario = factor(scenario, levels = paste0("S", 1:6)))
scenarios <- paste0("S", 1:6)
versions <- c("1.0.0", "1.1.0", "1.2.0")
# Expected condition (1 = satisfied) by scenario and dimension, from
# the specification, section 10. These are validation fixtures only;
# they never enter a prompt.
expected_condition <- matrix(
c(
0.85, 0.90, 0.80, 0.90, 0.70, 0.40, 0.30,
0.40, 0.70, 0.50, 0.70, 0.40, 0.30, 0.20,
0.90, 0.45, 0.50, 0.40, 0.55, 0.20, 0.15,
0.40, 0.40, 0.40, 0.30, 0.30, 0.20, 0.20,
0.30, 0.30, 0.30, 0.40, 0.30, 0.20, 0.10,
0.20, 0.20, 0.20, 0.10, 0.20, 0.20, 0.20
),
nrow = 6,
byrow = TRUE,
dimnames = list(scenarios, liminal2::liminal_dimensions()$name)
)
expected <- as.data.frame(as.table(expected_condition)) |>
tibble::as_tibble() |>
dplyr::rename(
scenario = Var1,
dimension = Var2,
condition = Freq
) |>
dplyr::mutate(
scenario = factor(scenario, levels = scenarios),
expected_raw = 10 * (1 - condition)
)
expected_L <- expected |>
dplyr::group_by(scenario) |>
dplyr::summarise(
expected_L = 1 - prod(condition^(1 / dplyr::n())),
.groups = "drop"
)
observed <- results |>
dplyr::group_by(version, model, scenario, dimension) |>
dplyr::summarise(raw = mean(raw_score), .groups = "drop")
liminality <- results |>
dplyr::group_by(version, model, scenario, rep) |>
dplyr::summarise(
L = liminal2::measure_liminal_space(
stats::setNames(1 - raw_score / 10, dimension)
),
.groups = "drop"
)The 1.0.0 liminality, averaged over the two repetitions,
was:
| version | model | S1 | S2 | S3 | S4 | S5 | S6 |
|---|---|---|---|---|---|---|---|
| 1.0.0 | deepseek-v4.1-flash | 0.72 | 0.71 | 0.67 | 0.83 | 0.72 | 1.00 |
| 1.0.0 | glm-5.3 | 0.67 | 0.67 | 0.57 | 0.81 | 0.76 | 0.80 |
| 1.0.0 | glm-5.3-flash | 0.70 | 0.68 | 0.43 | 0.77 | 0.74 | 0.88 |
The problem is not a single bad dimension. It is dimension bleed: each prompt gives anchors but no instruction about which evidence belongs to the dimension, so the model absorbs evidence that belongs elsewhere.
-
shared_meaningabsorbed the performance horizon, the intended use and output granularity, and treated any ambiguity as rival definitions. -
action_identifiabletreated a named verb (“block fraud”, “set prices”) as a complete decision rule, even without a threshold, an owner or a cost of errors. -
knowable_from_observablesmixed coverage and join gaps with selection and censoring. -
judgeableread a vague ask as “no independent adjudicator”. -
convergent_answersasked for an unobservable quantity (the share of decisions that flip among near-best models), so the model guessed, and its guesses were compressed toward the middle. -
stable_groundwas the exception: it tracked the expected ordering closely and was left essentially unchanged. -
tradeoff_agreedwas accurate but nearly constant across scenarios, so it contributed little to the ordering.
The lesson is that anchors alone are not a scoring procedure. They define the endpoints of the scale, but not how to read a messy case that sits between two anchors, nor which evidence the dimension owns.
Stage 2 (1.1.0): scope guards and scoring
procedures
Version 1.1.0 adds two sections to every dimension
template, leaving the anchors, the direction statement and the output
contract unchanged:
-
A scope guard (“What counts”). It states which
evidence the dimension owns and, explicitly, which evidence it must not
absorb from another dimension. Example,
shared_meaning: count only rival definitions that change who is classified positive; do not count the horizon, the intended use, the output granularity, the absence of a baseline, or feature availability. - An ordered scoring procedure (“How to score”). It makes the mapping from evidence to score mechanical: quote the relevant evidence, match it to the closest anchor, apply the dimension’s partial-credit rules, interpolate, and reserve the extreme scores for genuinely structural cases.
The partial-credit rules are the part that changed behaviour most.
Some of them were also too strong. For action_identifiable,
for example, a named action that lacks any of the four parts (who,
threshold, timing, what changes) was at most a 5, but no threshold, no
owner or no timing forced at least a 6, and no cost of each error forced
at least a 7. Those floors contradicted the anchors and double-counted
tradeoff_agreed; the next stage removes them. The table
below records 1.1.0 as it was, not as the current
design.
| Dimension |
1.0.0 problem |
1.1.0 scope guard / procedure |
|---|---|---|
shared_meaning |
Any ambiguity counted as rival definitions | Count only rival definitions that change who is positive; cannot exceed 3 without a quotable second definition; explicitly settled labels cap the score at 4 |
stable_ground |
Could absorb selection and censoring | Stability only; selection, censoring and missing labels are
explicitly routed to knowable_from_observables
|
convergent_answers |
Unobservable evidence, compressed guesses | Enumerate defensible target definitions and model families; 2/5/8 map to 1/2/3+ combinations; no score below 4 unless both are pinned down |
knowable_from_observables |
Mixed coverage, selection and latent causes | Selection, truncation, censoring and post-hoc outcomes each force at least 6; coverage and join gaps count |
judgeable |
Vague ask read as no adjudicator | Adjudicator is independent of the ask’s clarity; an implied auditor or regulator caps the score at 4 even when the ask is vague |
action_identifiable |
A named verb counted as a decision rule | Four-part test; partial credit for each missing part; no cost of errors forces at least 7 |
tradeoff_agreed |
Missing cost matrix read as agreement | A missing cost matrix is evidence of an unresolved trade-off and forces at least 6 |
The anchor scores themselves did not move. That matters: the change
sharpens which anchor a case is matched to, it does not
redefine the scale. Scores from 1.0.0, 1.1.0
and 1.2.0 are therefore not interchangeable, and the
package records the prompt version in every result and every benchmark
row.
Why not ask the model to recall other organizations?
A tempting alternative is to ground each call by asking the model for “similar experiences in other organizations”. We deliberately did not do that, for four reasons:
- It grounds the score in the model’s memory, not in the ask. The score would depend on which clients a given model happens to have seen in training, which is unauditable and differs between models.
- It breaks comparability. Two models with different training corpora would no longer be measuring the same thing, violating the comparability contract.
- It compresses toward the prior. “Most organizations sit in the middle” pulls every case toward the same score, which is the opposite of what a measure meant to separate cases needs.
-
It does not address the actual failure. The
1.0.0benchmark failed on ordering and on dimension bleed, not on a global level offset. A reference-class prior moves levels; it does not stopshared_meaningfrom absorbing the horizon.
The auditable form of a reference class is the fixed anchor rubric,
which is already in every prompt. 1.1.0 makes the model use
it consistently instead of replacing it with recollection.
Stage 3 (1.2.0): remove the floors, calibrate the
structure
The 1.1.0 benchmark showed that the scope guards fixed
the dimensions they targeted, but several rules were over-specified. The
model followed those rules exactly, which is why the scores still
clustered at 7 and 8. 1.2.0 is a definition-driven
correction, not a fit to the ladder.
| Dimension |
1.1.0 defect |
1.2.0 correction |
|---|---|---|
| all | The 8 anchor was treated as the default for any ambiguity | Match the anchor’s structural condition, not the amount of missing information; 8 is not a default, and 7 is an interpolation between 5 and 8 |
shared_meaning |
An undefined normative word was treated as a rival label definition | A descriptive ask with named quantities scores 1-2; multiple positive classes under one label score 6-8 only when they change who is positive |
stable_ground |
Selection, censoring and stockouts were routed exclusively to
knowable; an invented “18-24 months” heuristic overrode the
evidence |
Policy feedback, model-dependent sampling and adversarial drift are structural breaks (5-9); the numeric heuristic is removed |
convergent_answers |
Counted defensible target definitions and method families, then forbade scores below 4 | Score the share of decisions or rankings that flip; method count alone is not disagreement, and there is no floor |
knowable_from_observables |
Absorbed undefined target meaning and the absence of a baseline | Undefined meaning is shared_meaning; complete, timely
descriptive data score 1-2 |
judgeable |
Treated regulated contexts and individual outcome labels as adjudication of the solution | Chargebacks and audits that label cases are not a rubric for the ask; a live policy with no success criterion scores 8-9, while a recomputable indicator scores 2-3 |
action_identifiable |
Contradictory “at most 5” and “at least 6” rules, plus a “no cost
matrix means at least 7” floor that double-counted
tradeoff
|
Four-part test, graded 0-9; cost belongs to tradeoff;
an implied monitoring use scores 6-7 and a pure information request
8-9 |
tradeoff_agreed |
A missing cost matrix forced at least 6, so every case looked unresolved | Missing matrix scores 7-8; a live decision with no matrix scores 8; a fixed capacity that silently sets the threshold scores 9; a descriptive ask with no operating decision scores 6-7 |
The anchors and the direction statement did not move.
1.2.0 changes only how evidence is read against the
existing anchors, so its scores are not interchangeable with
1.0.0 or 1.1.0.
We made this revision once, because each defect is visible in the prompt text against the dimension definition. We did not tune a rule to a particular scenario, and we did not iterate after the full benchmark. A small S1/S5 probe was used only to confirm that the revised prompts were functioning before the 252-call run; it was not a fitting target.
What changed in the measurements
Per-dimension fit
The table below compares each version’s scores with the validation
fixtures. MAE is the mean absolute error on the 0–10 raw
scale, Spearman is the rank correlation across the six
scenarios, and spread is the standard deviation of the six
scenario means. A good dimension has a low MAE, a high rank correlation
and a spread close to the fixture’s.
| dimension | MAE_1.0.0 | MAE_1.1.0 | MAE_1.2.0 | Spearman_1.0.0 | Spearman_1.1.0 | Spearman_1.2.0 | spread_1.0.0 | spread_1.1.0 | spread_1.2.0 |
|---|---|---|---|---|---|---|---|---|---|
| shared_meaning | 2.94 | 2.33 | 2.47 | 0.41 | 0.32 | 0.09 | 1.72 | 2.34 | 2.74 |
| stable_ground | 0.58 | 1.58 | 0.64 | 0.93 | 0.84 | 0.89 | 2.55 | 2.29 | 2.60 |
| convergent_answers | 2.44 | 2.83 | 1.08 | 0.84 | 0.84 | 0.46 | 1.03 | 0.41 | 2.05 |
| knowable_from_observables | 1.58 | 1.56 | 0.81 | 0.70 | 0.93 | 0.99 | 1.97 | 1.91 | 2.48 |
| judgeable | 1.17 | 1.64 | 1.19 | 0.66 | 0.64 | 0.72 | 1.53 | 2.29 | 2.26 |
| action_identifiable | 1.83 | 0.89 | 1.33 | -0.36 | -0.47 | -0.22 | 1.54 | 0.54 | 1.05 |
| tradeoff_agreed | 0.67 | 0.69 | 0.42 | 0.10 | 0.25 | 0.89 | 0.56 | 0.33 | 0.92 |
The scenario ladder
| version | model | inversions | S1 | S2 | S3 | S4 | S5 | S6 |
|---|---|---|---|---|---|---|---|---|
| 1.0.0 | deepseek-v4.1-flash | 3 | 0.72 | 0.71 | 0.67 | 0.83 | 0.72 | 1.00 |
| 1.0.0 | glm-5.3 | 3 | 0.67 | 0.67 | 0.57 | 0.81 | 0.76 | 0.80 |
| 1.0.0 | glm-5.3-flash | 3 | 0.70 | 0.68 | 0.43 | 0.77 | 0.74 | 0.88 |
| 1.1.0 | deepseek-v4.1-flash | 2 | 0.66 | 0.72 | 0.58 | 0.78 | 0.63 | 1.00 |
| 1.1.0 | glm-5.3 | 2 | 0.62 | 0.67 | 0.55 | 0.76 | 0.71 | 0.78 |
| 1.1.0 | glm-5.3-flash | 3 | 0.62 | 0.66 | 0.55 | 0.67 | 0.62 | 0.88 |
| 1.2.0 | deepseek-v4.1-flash | 1 | 0.36 | 0.71 | 0.71 | 0.72 | 0.75 | 0.89 |
| 1.2.0 | glm-5.3 | 3 | 0.38 | 0.70 | 0.65 | 0.62 | 0.79 | 0.72 |
| 1.2.0 | glm-5.3-flash | 3 | 0.40 | 0.66 | 0.61 | 0.58 | 0.75 | 0.87 |
Summary
| version | Spearman_L | inverted_steps | models_with_ladder | raw_jitter | L_jitter | total_cost |
|---|---|---|---|---|---|---|
| 1.0.0 | 0.714 | 9 | 0 | 0.333 | 0.028 | 0.800 |
| 1.1.0 | 0.600 | 7 | 0 | 0.389 | 0.036 | 0.989 |
| 1.2.0 | 0.771 | 7 | 0 | 0.397 | 0.045 | 1.408 |
The number of inverted model-by-ladder steps was 9 under
1.0.0, 7 under 1.1.0 and 7 under
1.2.0, out of 15. The number of models reproducing the full
ladder was 0, 0 and 0, out of 3.
Per-dimension change
| dimension | MAE 1.0->1.1 | MAE 1.1->1.2 | Spearman 1.1->1.2 |
|---|---|---|---|
| convergent_answers | 0.39 | -1.75 | -0.37 |
| stable_ground | 1.00 | -0.94 | 0.05 |
| knowable_from_observables | -0.03 | -0.75 | 0.06 |
| judgeable | 0.47 | -0.44 | 0.08 |
| tradeoff_agreed | 0.03 | -0.28 | 0.64 |
| shared_meaning | -0.61 | 0.14 | -0.23 |
| action_identifiable | -0.94 | 0.44 | 0.25 |
Negative MAE 1.1->1.2 means the dimension moved
closer to the fixture. Relative to 1.1.0,
1.2.0 moved convergent_answers, stable_ground,
knowable_from_observables, judgeable, tradeoff_agreed closer to the
fixture and shared_meaning, action_identifiable further away; no
dimension was left unchanged.
Raw-score movement
| dimension | 1.0.0 | 1.1.0 | 1.2.0 | 1.0->1.1 | 1.1->1.2 |
|---|---|---|---|---|---|
| shared_meaning | 6.86 | 3.97 | 3.28 | -2.89 | -0.69 |
| stable_ground | 5.44 | 3.83 | 5.56 | -1.61 | 1.72 |
| convergent_answers | 7.94 | 8.33 | 6.42 | 0.39 | -1.92 |
| knowable_from_observables | 6.42 | 6.89 | 6.08 | 0.47 | -0.81 |
| judgeable | 6.86 | 5.39 | 6.44 | -1.47 | 1.06 |
| action_identifiable | 6.33 | 7.39 | 6.50 | 1.06 | -0.89 |
| tradeoff_agreed | 7.75 | 7.44 | 7.67 | -0.31 | 0.22 |
What the benchmark showed
The S1–S5 separation
The validation fixtures are designed to put S1 at the closed end and S5 at the open end. The table shows the model-averaged liminality for each prompt version, next to the fixture’s expected value.
| source | S1 | S5 | gap |
|---|---|---|---|
| fixture | 0.355 | 0.748 | 0.393 |
| 1.0.0 | 0.696 | 0.739 | 0.043 |
| 1.1.0 | 0.630 | 0.654 | 0.023 |
| 1.2.0 | 0.378 | 0.761 | 0.383 |
The 1.1.0 revision fixed dimension bleed but introduced
new floors. 1.2.0 removes them. The sections below separate
what each stage changed.
What 1.1.0 fixed
-
action_identifiableis the clearest win. The mean absolute error roughly halved. S5 and S6, which1.0.0scored as if “block fraud” and “set prices” were complete decision rules, moved up sharply once the four-part test made the missing threshold and owner explicit. -
shared_meaningmoved closer to the fixtures. S1, S3, S4 and S6 fell to 2–4, which is what the scope guard intends: with one stated definition and no quotable rival, the guard pulls the score toward the low anchor. S2 stayed high because it really does describe rival definitions. Part of the remaining gap is the fixture, not the model: S6 is expected to score 8 on shared meaning, yet its text describes no rival definition of the target. That is dimension bleed in the validation fixture. -
knowable_from_observablesheld its mean absolute error and its rank correlation rose to 0.93. S3, the approval-selected portfolio, moved into the expected range once selection was treated as a knowability problem.
What 1.1.0 broke
-
convergent_answerssaturated at 8-9 in every scenario. The rule “do not score below 4 unless both the target and the method are pinned down”, combined with the enumerate-the-families procedure, made almost every tabular task look like it had three defensible combinations. Its spread collapsed, so the dimension stopped discriminating between cases. -
stable_ground, previously the best dimension, regressed. The invented heuristic “18-24 months of comparable history maps to 2-3” overrode the gradual-drift evidence, and routing selection and censoring toknowable_from_observablesremoved penalties the fixture expects here. The routing rule is defensible on the definition; the numeric heuristic is not, because it is not in the anchors. -
judgeableregressed, mainly because “regulated contexts usually imply an external procedure” turned S5’s chargeback investigation into an adjudicator. That conflates confirming a fraud with adjudicating whether the solution satisfied the ask.
What 1.2.0 changed
Relative to 1.1.0, 1.2.0 moved
convergent_answers, stable_ground, knowable_from_observables, judgeable,
tradeoff_agreed closer to the fixture, and moved shared_meaning,
action_identifiable away from it. The point is not that every dimension
improved; it is that the rules now read the existing anchors instead of
overriding them. The one substantive cost is
action_identifiable: S5 and S6 both score below the
fixture’s 8, and S6 lands below the prompt’s own 6–7 band for a named
action that lacks a threshold and an owner. That is a remaining
calibration gap, not a reason to revise again in this iteration.
At the model-averaged level, S1 moved from 0.630 to 0.378, and S5 from 0.654 to 0.761. The S1–S5 gap is 0.383. The number of inverted model-by-ladder steps is 7 of 15, and 0 of 3 models reproduce the full ladder.
Cost rose from 0.80 under 1.0.0 to 0.99 under
1.1.0 and 1.41 under 1.2.0, because the
prompts are longer. Final-score jitter was 0.028, 0.036 and 0.045: the
extra instructions did not make the measurement materially less
stable.
Why we stop here
The 1.2.0 revision removed the rules that a close
reading of the prompts showed to be inconsistent with the dimension
definitions. It was a single, definition-driven correction; it was not
tuned to the ladder, and it is not followed by another iteration.
The remaining gaps are anchor-calibration evidence, not another prompt wording problem. The correct next step is the independent anchor review in specification section 14.1: raters who did not write the prompts, scoring blind cases outside this suite. Further tuning against S1–S6 would be fitting the prompt to the validation data, which is exactly the circularity the design forbids.
Limitations
- The six scenarios are validation data. They can reveal a defect, but they cannot certify a fix; a prompt that reproduces this particular ladder may still be wrong on the next business case.
- A monotone ladder is a strong claim for a product of seven noisy
integers. S2 versus S3 is a genuine disagreement about the case, not
only a wording defect, and under
1.2.0the middle of the ladder (S2, S3, S4) is effectively flat, so the monotone claim is not met. - One fixture expectation disagrees with the anchors. S6’s shared
meaning expects 8, yet the scenario names a single target (price
elasticity) and no rival definition, so the dimension should be low;
that mismatch is a sizeable part of
shared_meaning’s mean absolute error. The target is correct dimension semantics, not exact reproduction of every fixture number. - Temperature 0 is not bitwise determinism on this endpoint; the jitter columns bound the variation, and repeats are not identical.
- Scores from
1.0.0,1.1.0and1.2.0must not be pooled. They are separate measurements on different prompt versions, and the package records the version in every result and every benchmark row.
Reproducing the comparison
The comparison needs no network access: it reads the three committed caches.
results <- dplyr::bind_rows(
read_cache("model-comparison-results-v1.0.0.parquet", "1.0.0"),
read_cache("model-comparison-results-v1.1.0.parquet", "1.1.0"),
read_cache("model-comparison-results.parquet", "1.2.0")
)To regenerate the current cache, run
data-raw/model_comparison.R with
PANAMA_API_KEY set. The script discovers the model list
live from the endpoint, records the prompt version on every row, and
writes inst/extdata/model-comparison-results.parquet. To
keep a version for comparison, copy the current cache to a versioned
filename before regenerating, exactly as was done for 1.0.0
and 1.1.0.