Skip to contents
library(liminal2)
library(arrow)
#> 
#> Attaching package: 'arrow'
#> The following object is masked from 'package:utils':
#> 
#>     timestamp
library(dplyr)
#> 
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#> 
#>     filter, lag
#> The following objects are masked from 'package:base':
#> 
#>     intersect, setdiff, setequal, union
library(tidyr)

Purpose

This vignette explains how the scoring method evolved and what the changes did to the measurements. It is written for users who want to understand why the package does not simply ask a model “how big is the liminal space?” and trust the answer.

The story has four stages:

  1. a naive single-shot judge, which is the obvious design and the one we deliberately did not ship;
  2. version 1.0.0, which decomposed the score into seven anchored, independently scored dimensions;
  3. version 1.1.0, which added a scope guard and an ordered scoring procedure to every dimension prompt;
  4. version 1.2.0, which removed the hard floors and cross-dimension double counting that 1.1.0 introduced, added a structural calibration rule, and sharpened the procedures that had conflated method uncertainty with decision disagreement.

All three prompt versions were benchmarked over the same six scenarios, the same three models and the same two repetitions. The 1.0.0 and 1.1.0 caches are frozen in inst/extdata/model-comparison-results-v1.0.0.parquet and inst/extdata/model-comparison-results-v1.1.0.parquet; the current cache is inst/extdata/model-comparison-results.parquet. All three ship with the package, so this vignette renders offline and the comparison is reproducible.

Stage 0: the naive judge

The naive design is one call to a language model with a prompt like:

“How large is the liminal space between this business ask and a data-based solution? Answer from 0 to 10.”

It is cheap to write and it fails in five distinct ways.

Failure mode What goes wrong
No decomposition Seven different obstacles collapse into one number. A large score cannot say whether the ask is ambiguous, the data are unsuitable, or the decision rule is missing, so it cannot guide any action.
No fixed scale Nothing defines what 0 and 10 mean, whether 0 is good or bad, or how this case compares with the next one. Two runs on the same ask can land on either side of the midpoint.
No reference class The model has no common yardstick across business cases. Scores drift with prompt wording, model version and the order of the conversation.
Free-text output The answer arrives as prose (“around a 6 or 7, depending on the data”), so it must be parsed heuristically. Parse failures and silent coercions become measurement error.
No provenance The result records neither the model, the prompt text, nor the scale, so two scores cannot be known to be comparable.

None of these are model failures. They are measurement-design failures: the naive judge produces an opinion, not a measurement. We did not benchmark this stage, because there is nothing stable to benchmark.

Stage 1 (1.0.0): decompose, anchor, constrain

Version 1.0.0 turned the opinion into a measurement with six design decisions:

  1. One prompt per dimension. Seven independent calls, each scoring exactly one of the dimensions in liminal_dimensions(). A failure or ambiguity in one dimension cannot contaminate the others.
  2. A fixed direction statement. Every prompt states, in identical words, that 0 is the smallest liminal space and 10 the largest, and that the answer must be an integer.
  3. Fixed anchor cases. Every dimension has three generic anchors at scores 2, 5 and 8. They describe structurally distinct situations, not named projects, and are versioned with the prompt.
  4. Structured output. The wire schema is a single integer field, score, so there is no free-text parsing.
  5. Determinism controls. Temperature 0, a fixed default model and stateless calls. (The gateway is not bitwise deterministic even at temperature 0; the tests bound the jitter rather than demanding identical repeats.)
  6. Provenance and anti-circularity. The result records the model, base URL and prompt versions. The six validation scenarios are used only to check the ordering of the final measure, never to write or tune prompts.

The aggregation is the geometric mean implied by the product rule:

L=1−∏i=17ciwi, L = 1 - \prod_{i=1}^{7} c_i^{w_i},

where ci=1−scorei/10c_i = 1 - \text{score}_i / 10 is the condition score of dimension ii and wiw_i its weight.

1.0.0 was benchmarked with 252 calls (three models, six scenarios, seven dimensions, two repetitions). Every call returned a valid score; no retry was needed. But the benchmark also exposed a systematic problem.

What 1.0.0 still got wrong

The scenarios are designed so that liminality rises from S1 to S6. In the 1.0.0 run it did not:

read_cache <- function(file, prompt_version) {
  arrow::read_parquet(
    system.file("extdata", file, package = "liminal2")
  ) |>
    mutate(version = .env$prompt_version)
}

results <- dplyr::bind_rows(
  read_cache("model-comparison-results-v1.0.0.parquet", "1.0.0"),
  read_cache("model-comparison-results-v1.1.0.parquet", "1.1.0"),
  read_cache("model-comparison-results.parquet", "1.2.0")
) |>
  mutate(scenario = factor(scenario, levels = paste0("S", 1:6)))

scenarios <- paste0("S", 1:6)
versions <- c("1.0.0", "1.1.0", "1.2.0")

# Expected condition (1 = satisfied) by scenario and dimension, from
# the specification, section 10. These are validation fixtures only;
# they never enter a prompt.
expected_condition <- matrix(
  c(
    0.85, 0.90, 0.80, 0.90, 0.70, 0.40, 0.30,
    0.40, 0.70, 0.50, 0.70, 0.40, 0.30, 0.20,
    0.90, 0.45, 0.50, 0.40, 0.55, 0.20, 0.15,
    0.40, 0.40, 0.40, 0.30, 0.30, 0.20, 0.20,
    0.30, 0.30, 0.30, 0.40, 0.30, 0.20, 0.10,
    0.20, 0.20, 0.20, 0.10, 0.20, 0.20, 0.20
  ),
  nrow = 6,
  byrow = TRUE,
  dimnames = list(scenarios, liminal2::liminal_dimensions()$name)
)

expected <- as.data.frame(as.table(expected_condition)) |>
  tibble::as_tibble() |>
  dplyr::rename(
    scenario = Var1,
    dimension = Var2,
    condition = Freq
  ) |>
  dplyr::mutate(
    scenario = factor(scenario, levels = scenarios),
    expected_raw = 10 * (1 - condition)
  )

expected_L <- expected |>
  dplyr::group_by(scenario) |>
  dplyr::summarise(
    expected_L = 1 - prod(condition^(1 / dplyr::n())),
    .groups = "drop"
  )

observed <- results |>
  dplyr::group_by(version, model, scenario, dimension) |>
  dplyr::summarise(raw = mean(raw_score), .groups = "drop")

liminality <- results |>
  dplyr::group_by(version, model, scenario, rep) |>
  dplyr::summarise(
    L = liminal2::measure_liminal_space(
      stats::setNames(1 - raw_score / 10, dimension)
    ),
    .groups = "drop"
  )

The 1.0.0 liminality, averaged over the two repetitions, was:

version model S1 S2 S3 S4 S5 S6
1.0.0 deepseek-v4.1-flash 0.72 0.71 0.67 0.83 0.72 1.00
1.0.0 glm-5.3 0.67 0.67 0.57 0.81 0.76 0.80
1.0.0 glm-5.3-flash 0.70 0.68 0.43 0.77 0.74 0.88

The problem is not a single bad dimension. It is dimension bleed: each prompt gives anchors but no instruction about which evidence belongs to the dimension, so the model absorbs evidence that belongs elsewhere.

  • shared_meaning absorbed the performance horizon, the intended use and output granularity, and treated any ambiguity as rival definitions.
  • action_identifiable treated a named verb (“block fraud”, “set prices”) as a complete decision rule, even without a threshold, an owner or a cost of errors.
  • knowable_from_observables mixed coverage and join gaps with selection and censoring.
  • judgeable read a vague ask as “no independent adjudicator”.
  • convergent_answers asked for an unobservable quantity (the share of decisions that flip among near-best models), so the model guessed, and its guesses were compressed toward the middle.
  • stable_ground was the exception: it tracked the expected ordering closely and was left essentially unchanged.
  • tradeoff_agreed was accurate but nearly constant across scenarios, so it contributed little to the ordering.

The lesson is that anchors alone are not a scoring procedure. They define the endpoints of the scale, but not how to read a messy case that sits between two anchors, nor which evidence the dimension owns.

Stage 2 (1.1.0): scope guards and scoring procedures

Version 1.1.0 adds two sections to every dimension template, leaving the anchors, the direction statement and the output contract unchanged:

  1. A scope guard (“What counts”). It states which evidence the dimension owns and, explicitly, which evidence it must not absorb from another dimension. Example, shared_meaning: count only rival definitions that change who is classified positive; do not count the horizon, the intended use, the output granularity, the absence of a baseline, or feature availability.
  2. An ordered scoring procedure (“How to score”). It makes the mapping from evidence to score mechanical: quote the relevant evidence, match it to the closest anchor, apply the dimension’s partial-credit rules, interpolate, and reserve the extreme scores for genuinely structural cases.

The partial-credit rules are the part that changed behaviour most. Some of them were also too strong. For action_identifiable, for example, a named action that lacks any of the four parts (who, threshold, timing, what changes) was at most a 5, but no threshold, no owner or no timing forced at least a 6, and no cost of each error forced at least a 7. Those floors contradicted the anchors and double-counted tradeoff_agreed; the next stage removes them. The table below records 1.1.0 as it was, not as the current design.

Dimension 1.0.0 problem 1.1.0 scope guard / procedure
shared_meaning Any ambiguity counted as rival definitions Count only rival definitions that change who is positive; cannot exceed 3 without a quotable second definition; explicitly settled labels cap the score at 4
stable_ground Could absorb selection and censoring Stability only; selection, censoring and missing labels are explicitly routed to knowable_from_observables
convergent_answers Unobservable evidence, compressed guesses Enumerate defensible target definitions and model families; 2/5/8 map to 1/2/3+ combinations; no score below 4 unless both are pinned down
knowable_from_observables Mixed coverage, selection and latent causes Selection, truncation, censoring and post-hoc outcomes each force at least 6; coverage and join gaps count
judgeable Vague ask read as no adjudicator Adjudicator is independent of the ask’s clarity; an implied auditor or regulator caps the score at 4 even when the ask is vague
action_identifiable A named verb counted as a decision rule Four-part test; partial credit for each missing part; no cost of errors forces at least 7
tradeoff_agreed Missing cost matrix read as agreement A missing cost matrix is evidence of an unresolved trade-off and forces at least 6

The anchor scores themselves did not move. That matters: the change sharpens which anchor a case is matched to, it does not redefine the scale. Scores from 1.0.0, 1.1.0 and 1.2.0 are therefore not interchangeable, and the package records the prompt version in every result and every benchmark row.

Why not ask the model to recall other organizations?

A tempting alternative is to ground each call by asking the model for “similar experiences in other organizations”. We deliberately did not do that, for four reasons:

  1. It grounds the score in the model’s memory, not in the ask. The score would depend on which clients a given model happens to have seen in training, which is unauditable and differs between models.
  2. It breaks comparability. Two models with different training corpora would no longer be measuring the same thing, violating the comparability contract.
  3. It compresses toward the prior. “Most organizations sit in the middle” pulls every case toward the same score, which is the opposite of what a measure meant to separate cases needs.
  4. It does not address the actual failure. The 1.0.0 benchmark failed on ordering and on dimension bleed, not on a global level offset. A reference-class prior moves levels; it does not stop shared_meaning from absorbing the horizon.

The auditable form of a reference class is the fixed anchor rubric, which is already in every prompt. 1.1.0 makes the model use it consistently instead of replacing it with recollection.

Stage 3 (1.2.0): remove the floors, calibrate the structure

The 1.1.0 benchmark showed that the scope guards fixed the dimensions they targeted, but several rules were over-specified. The model followed those rules exactly, which is why the scores still clustered at 7 and 8. 1.2.0 is a definition-driven correction, not a fit to the ladder.

Dimension 1.1.0 defect 1.2.0 correction
all The 8 anchor was treated as the default for any ambiguity Match the anchor’s structural condition, not the amount of missing information; 8 is not a default, and 7 is an interpolation between 5 and 8
shared_meaning An undefined normative word was treated as a rival label definition A descriptive ask with named quantities scores 1-2; multiple positive classes under one label score 6-8 only when they change who is positive
stable_ground Selection, censoring and stockouts were routed exclusively to knowable; an invented “18-24 months” heuristic overrode the evidence Policy feedback, model-dependent sampling and adversarial drift are structural breaks (5-9); the numeric heuristic is removed
convergent_answers Counted defensible target definitions and method families, then forbade scores below 4 Score the share of decisions or rankings that flip; method count alone is not disagreement, and there is no floor
knowable_from_observables Absorbed undefined target meaning and the absence of a baseline Undefined meaning is shared_meaning; complete, timely descriptive data score 1-2
judgeable Treated regulated contexts and individual outcome labels as adjudication of the solution Chargebacks and audits that label cases are not a rubric for the ask; a live policy with no success criterion scores 8-9, while a recomputable indicator scores 2-3
action_identifiable Contradictory “at most 5” and “at least 6” rules, plus a “no cost matrix means at least 7” floor that double-counted tradeoff Four-part test, graded 0-9; cost belongs to tradeoff; an implied monitoring use scores 6-7 and a pure information request 8-9
tradeoff_agreed A missing cost matrix forced at least 6, so every case looked unresolved Missing matrix scores 7-8; a live decision with no matrix scores 8; a fixed capacity that silently sets the threshold scores 9; a descriptive ask with no operating decision scores 6-7

The anchors and the direction statement did not move. 1.2.0 changes only how evidence is read against the existing anchors, so its scores are not interchangeable with 1.0.0 or 1.1.0.

We made this revision once, because each defect is visible in the prompt text against the dimension definition. We did not tune a rule to a particular scenario, and we did not iterate after the full benchmark. A small S1/S5 probe was used only to confirm that the revised prompts were functioning before the 252-call run; it was not a fitting target.

What changed in the measurements

Per-dimension fit

The table below compares each version’s scores with the validation fixtures. MAE is the mean absolute error on the 0–10 raw scale, Spearman is the rank correlation across the six scenarios, and spread is the standard deviation of the six scenario means. A good dimension has a low MAE, a high rank correlation and a spread close to the fixture’s.

dimension MAE_1.0.0 MAE_1.1.0 MAE_1.2.0 Spearman_1.0.0 Spearman_1.1.0 Spearman_1.2.0 spread_1.0.0 spread_1.1.0 spread_1.2.0
shared_meaning 2.94 2.33 2.47 0.41 0.32 0.09 1.72 2.34 2.74
stable_ground 0.58 1.58 0.64 0.93 0.84 0.89 2.55 2.29 2.60
convergent_answers 2.44 2.83 1.08 0.84 0.84 0.46 1.03 0.41 2.05
knowable_from_observables 1.58 1.56 0.81 0.70 0.93 0.99 1.97 1.91 2.48
judgeable 1.17 1.64 1.19 0.66 0.64 0.72 1.53 2.29 2.26
action_identifiable 1.83 0.89 1.33 -0.36 -0.47 -0.22 1.54 0.54 1.05
tradeoff_agreed 0.67 0.69 0.42 0.10 0.25 0.89 0.56 0.33 0.92

The scenario ladder

version model inversions S1 S2 S3 S4 S5 S6
1.0.0 deepseek-v4.1-flash 3 0.72 0.71 0.67 0.83 0.72 1.00
1.0.0 glm-5.3 3 0.67 0.67 0.57 0.81 0.76 0.80
1.0.0 glm-5.3-flash 3 0.70 0.68 0.43 0.77 0.74 0.88
1.1.0 deepseek-v4.1-flash 2 0.66 0.72 0.58 0.78 0.63 1.00
1.1.0 glm-5.3 2 0.62 0.67 0.55 0.76 0.71 0.78
1.1.0 glm-5.3-flash 3 0.62 0.66 0.55 0.67 0.62 0.88
1.2.0 deepseek-v4.1-flash 1 0.36 0.71 0.71 0.72 0.75 0.89
1.2.0 glm-5.3 3 0.38 0.70 0.65 0.62 0.79 0.72
1.2.0 glm-5.3-flash 3 0.40 0.66 0.61 0.58 0.75 0.87

Summary

version Spearman_L inverted_steps models_with_ladder raw_jitter L_jitter total_cost
1.0.0 0.714 9 0 0.333 0.028 0.800
1.1.0 0.600 7 0 0.389 0.036 0.989
1.2.0 0.771 7 0 0.397 0.045 1.408

The number of inverted model-by-ladder steps was 9 under 1.0.0, 7 under 1.1.0 and 7 under 1.2.0, out of 15. The number of models reproducing the full ladder was 0, 0 and 0, out of 3.

Per-dimension change

dimension MAE 1.0->1.1 MAE 1.1->1.2 Spearman 1.1->1.2
convergent_answers 0.39 -1.75 -0.37
stable_ground 1.00 -0.94 0.05
knowable_from_observables -0.03 -0.75 0.06
judgeable 0.47 -0.44 0.08
tradeoff_agreed 0.03 -0.28 0.64
shared_meaning -0.61 0.14 -0.23
action_identifiable -0.94 0.44 0.25

Negative MAE 1.1->1.2 means the dimension moved closer to the fixture. Relative to 1.1.0, 1.2.0 moved convergent_answers, stable_ground, knowable_from_observables, judgeable, tradeoff_agreed closer to the fixture and shared_meaning, action_identifiable further away; no dimension was left unchanged.

Raw-score movement

dimension 1.0.0 1.1.0 1.2.0 1.0->1.1 1.1->1.2
shared_meaning 6.86 3.97 3.28 -2.89 -0.69
stable_ground 5.44 3.83 5.56 -1.61 1.72
convergent_answers 7.94 8.33 6.42 0.39 -1.92
knowable_from_observables 6.42 6.89 6.08 0.47 -0.81
judgeable 6.86 5.39 6.44 -1.47 1.06
action_identifiable 6.33 7.39 6.50 1.06 -0.89
tradeoff_agreed 7.75 7.44 7.67 -0.31 0.22

What the benchmark showed

The S1–S5 separation

The validation fixtures are designed to put S1 at the closed end and S5 at the open end. The table shows the model-averaged liminality for each prompt version, next to the fixture’s expected value.

source S1 S5 gap
fixture 0.355 0.748 0.393
1.0.0 0.696 0.739 0.043
1.1.0 0.630 0.654 0.023
1.2.0 0.378 0.761 0.383

The 1.1.0 revision fixed dimension bleed but introduced new floors. 1.2.0 removes them. The sections below separate what each stage changed.

What 1.1.0 fixed

  • action_identifiable is the clearest win. The mean absolute error roughly halved. S5 and S6, which 1.0.0 scored as if “block fraud” and “set prices” were complete decision rules, moved up sharply once the four-part test made the missing threshold and owner explicit.
  • shared_meaning moved closer to the fixtures. S1, S3, S4 and S6 fell to 2–4, which is what the scope guard intends: with one stated definition and no quotable rival, the guard pulls the score toward the low anchor. S2 stayed high because it really does describe rival definitions. Part of the remaining gap is the fixture, not the model: S6 is expected to score 8 on shared meaning, yet its text describes no rival definition of the target. That is dimension bleed in the validation fixture.
  • knowable_from_observables held its mean absolute error and its rank correlation rose to 0.93. S3, the approval-selected portfolio, moved into the expected range once selection was treated as a knowability problem.

What 1.1.0 broke

  • convergent_answers saturated at 8-9 in every scenario. The rule “do not score below 4 unless both the target and the method are pinned down”, combined with the enumerate-the-families procedure, made almost every tabular task look like it had three defensible combinations. Its spread collapsed, so the dimension stopped discriminating between cases.
  • stable_ground, previously the best dimension, regressed. The invented heuristic “18-24 months of comparable history maps to 2-3” overrode the gradual-drift evidence, and routing selection and censoring to knowable_from_observables removed penalties the fixture expects here. The routing rule is defensible on the definition; the numeric heuristic is not, because it is not in the anchors.
  • judgeable regressed, mainly because “regulated contexts usually imply an external procedure” turned S5’s chargeback investigation into an adjudicator. That conflates confirming a fraud with adjudicating whether the solution satisfied the ask.

What 1.2.0 changed

Relative to 1.1.0, 1.2.0 moved convergent_answers, stable_ground, knowable_from_observables, judgeable, tradeoff_agreed closer to the fixture, and moved shared_meaning, action_identifiable away from it. The point is not that every dimension improved; it is that the rules now read the existing anchors instead of overriding them. The one substantive cost is action_identifiable: S5 and S6 both score below the fixture’s 8, and S6 lands below the prompt’s own 6–7 band for a named action that lacks a threshold and an owner. That is a remaining calibration gap, not a reason to revise again in this iteration.

At the model-averaged level, S1 moved from 0.630 to 0.378, and S5 from 0.654 to 0.761. The S1–S5 gap is 0.383. The number of inverted model-by-ladder steps is 7 of 15, and 0 of 3 models reproduce the full ladder.

Cost rose from 0.80 under 1.0.0 to 0.99 under 1.1.0 and 1.41 under 1.2.0, because the prompts are longer. Final-score jitter was 0.028, 0.036 and 0.045: the extra instructions did not make the measurement materially less stable.

Why we stop here

The 1.2.0 revision removed the rules that a close reading of the prompts showed to be inconsistent with the dimension definitions. It was a single, definition-driven correction; it was not tuned to the ladder, and it is not followed by another iteration.

The remaining gaps are anchor-calibration evidence, not another prompt wording problem. The correct next step is the independent anchor review in specification section 14.1: raters who did not write the prompts, scoring blind cases outside this suite. Further tuning against S1–S6 would be fitting the prompt to the validation data, which is exactly the circularity the design forbids.

Limitations

  • The six scenarios are validation data. They can reveal a defect, but they cannot certify a fix; a prompt that reproduces this particular ladder may still be wrong on the next business case.
  • A monotone ladder is a strong claim for a product of seven noisy integers. S2 versus S3 is a genuine disagreement about the case, not only a wording defect, and under 1.2.0 the middle of the ladder (S2, S3, S4) is effectively flat, so the monotone claim is not met.
  • One fixture expectation disagrees with the anchors. S6’s shared meaning expects 8, yet the scenario names a single target (price elasticity) and no rival definition, so the dimension should be low; that mismatch is a sizeable part of shared_meaning’s mean absolute error. The target is correct dimension semantics, not exact reproduction of every fixture number.
  • Temperature 0 is not bitwise determinism on this endpoint; the jitter columns bound the variation, and repeats are not identical.
  • Scores from 1.0.0, 1.1.0 and 1.2.0 must not be pooled. They are separate measurements on different prompt versions, and the package records the version in every result and every benchmark row.

Reproducing the comparison

The comparison needs no network access: it reads the three committed caches.

results <- dplyr::bind_rows(
  read_cache("model-comparison-results-v1.0.0.parquet", "1.0.0"),
  read_cache("model-comparison-results-v1.1.0.parquet", "1.1.0"),
  read_cache("model-comparison-results.parquet", "1.2.0")
)

To regenerate the current cache, run data-raw/model_comparison.R with PANAMA_API_KEY set. The script discovers the model list live from the endpoint, records the prompt version on every row, and writes inst/extdata/model-comparison-results.parquet. To keep a version for comparison, copy the current cache to a versioned filename before regenerating, exactly as was done for 1.0.0 and 1.1.0.