
Model comparison
model-comparison.Rmd
library(liminal2)
library(arrow)
#>
#> Attaching package: 'arrow'
#> The following object is masked from 'package:utils':
#>
#> timestamp
library(dplyr)
#>
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#>
#> filter, lag
#> The following objects are masked from 'package:base':
#>
#> intersect, setdiff, setequal, union
library(tidyr)Purpose
This vignette benchmarks how the seven dimension scores and the final
liminality differ across all models available on the Panama
API, and presents the evidence in tables. It makes the choice
of the default model (glm-5.3) an auditable, visible
decision instead of an assertion.
The benchmark is evaluation, not calibration: the scenario ladder is used as a benchmark metric, never as a fitting target, and the results are never used to re-tune prompts or anchors. Scores from different models must not be pooled.
results <- arrow::read_parquet(
system.file(
"extdata",
"model-comparison-results.parquet",
package = "liminal2"
)
)
scenarios <- unique(results$scenario)
models <- unique(results$model)
benchmark_date <- unique(results$benchmark_date)The benchmark was regenerated from live API calls on 2026-11-10 under
prompt version 1.2.0, and covers 3 models (deepseek-v4.1-flash, glm-5.3,
glm-5.3-flash), the six scenarios S1–S6 and the seven dimension prompts,
with each cell run twice to quantify run-to-run stability at temperature
0. The raw scores, token usage and cost of every call are cached in
inst/extdata/model-comparison-results.parquet. The
superseded 1.0.0 and 1.1.0 runs are kept in
inst/extdata/model-comparison-results-v1.0.0.parquet and
inst/extdata/model-comparison-results-v1.1.0.parquet, and
the staged comparison is in the From a naive LLM judge to a rigorous
one vignette.
Liminality per model and scenario
Each row of the headline table aggregates the two repetitions. The
condition scores are computed from the raw scores with the fixed
conversion from the specification
(condition = 1 - raw / 10) and aggregated with
measure_liminal_space().
liminality <- results |>
group_by(model, scenario, rep) |>
summarise(
liminality = liminal2::measure_liminal_space(
stats::setNames(1 - raw_score / 10, dimension)
),
.groups = "drop"
)
headline <- liminality |>
group_by(model, scenario) |>
summarise(
mean_liminality = mean(liminality),
rep_delta = abs(diff(liminality)),
.groups = "drop"
) |>
mutate(scenario = factor(scenario, levels = scenarios)) |>
arrange(model, scenario)
ladder <- liminality |>
mutate(scenario = factor(scenario, levels = scenarios)) |>
arrange(model, rep, scenario) |>
group_by(model, rep) |>
summarise(ladder_ok = all(diff(liminality) > 0), .groups = "drop") |>
group_by(model) |>
summarise(ladder_ok = all(ladder_ok), .groups = "drop")
stability <- headline |>
group_by(model) |>
summarise(mean_delta = mean(rep_delta), .groups = "drop")
cost <- results |>
group_by(model) |>
summarise(total_cost = sum(cost, na.rm = TRUE), .groups = "drop")
# One row per model: the per-scenario rep_delta is summarised by
# mean_delta below, so it must not survive into the wide table (it
# would become a row identifier and shatter the pivot into mostly-NA
# rows).
table1 <- headline |>
mutate(mean_liminality = round(mean_liminality, 2)) |>
dplyr::select(-rep_delta) |>
pivot_wider(names_from = scenario, values_from = mean_liminality) |>
left_join(stability, by = "model") |>
left_join(ladder, by = "model") |>
left_join(cost, by = "model") |>
mutate(
mean_delta = round(mean_delta, 2),
total_cost = round(total_cost, 2)
)
knitr::kable(table1)| model | S1 | S2 | S3 | S4 | S5 | S6 | mean_delta | ladder_ok | total_cost |
|---|---|---|---|---|---|---|---|---|---|
| deepseek-v4.1-flash | 0.36 | 0.71 | 0.71 | 0.72 | 0.75 | 0.89 | 0.05 | FALSE | 0.1 |
| glm-5.3 | 0.38 | 0.70 | 0.65 | 0.62 | 0.79 | 0.72 | 0.02 | FALSE | 1.2 |
| glm-5.3-flash | 0.40 | 0.66 | 0.61 | 0.58 | 0.75 | 0.87 | 0.06 | FALSE | 0.1 |
ladder_ok is TRUE when the six liminalities
are strictly increasing along S1–S6 in both repetitions, the invariant
the measure must satisfy. mean_delta is the mean absolute
difference between the two repetitions across scenarios: the stability
of the model as a scorer. At temperature 0 it is often small but not
necessarily zero, because batched inference on the serving backend makes
even greedy sampling vary between runs; non-zero values quantify that
residual jitter. total_cost is summed over all calls of the
benchmark.
Reading the benchmark
n_ladder_ok <- sum(ladder$ladder_ok)
default_model <- "glm-5.3"
default_liminality <- liminality |>
filter(model == default_model) |>
group_by(scenario) |>
summarise(liminality = mean(liminality), .groups = "drop") |>
arrange(factor(scenario, levels = scenarios)) |>
pull(liminality)0 of 3 models reproduce the strictly increasing ladder. The default model, glm-5.3, yields the liminalities 0.38, 0.70, 0.65, 0.62, 0.79, 0.72 along S1–S6. When a model deviates, the dimension-level scores remain in the cached parquet, and the From a naive LLM judge to a rigorous one vignette examines which dimensions drive the difference: disagreement concentrated in one dimension is evidence about that dimension’s anchors, while diffuse disagreement is evidence about the model. Stability deltas above zero mean the model does not score deterministically even at temperature 0, and its scores should not be compared at a finer resolution than the observed jitter. Per the specification, such findings feed the anchor review; they are never used to re-tune the prompts on these scenarios.
Regenerate the cached results with: