When a language model is evaluated, the benchmark score is usually reported for one prompt format, or averaged over a handful of formats. Both practices quietly assume that every format measures the same underlying skill in every model. That assumption is not trivial. Suppose one model loses ten points whenever answers must be returned as JSON, and another model does not. Averaging over formats does not remove that difference. It mixes a model-specific formatting weakness into what we call the model's knowledge. Psychometrics has a standard test for this situation, called measurement invariance: before comparing two groups on a hidden trait, check whether each question relates to that trait in the same way in both groups. We propose to apply this test to LLM evaluation by flipping the usual setup. Benchmark questions play the role of test-takers. Prompt-perturbation families (whitespace padding, typos, paraphrase, option shuffling, symbol labels, JSON output, persona framing) play the role of questionnaire items. Each model is a separate group.
The core claim is that prompt brittleness comes in two kinds that current tools cannot tell apart. The first is random noise, which averaging fixes. The second is a model-specific, format-specific shift, which averaging cannot fix and which flips leaderboard rankings. A multi-group structural equation model separates the two and estimates each model's competence only from the formats that behave the same way across models. Structural equation modeling (SEM) is a statistical method that ties observed scores to unobserved traits while estimating measurement error explicitly. If the claim holds, scores built this way should rank models more stably on prompt formats never seen during fitting. They should also resist a planted, purely format-level defect that we inject into a model on purpose.
What exists and what is missing
The prompt-sensitivity literature establishes the problem but measures it with summary statistics.
Brittlebench applies semantics-preserving perturbations across MMLU, ARC, TruthfulQA, MathQA, LogiQA and GPQA. It splits total variance into an item-difficulty part and a brittleness part with an ANOVA-style decomposition. It finds that a single perturbation changes pairwise model rankings in 63% of cases. When Benchmarks are Targets shows that option shuffling and rare symbol labels move models up to 8 leaderboard positions. Beyond Benchmarks runs a factorial design over ten evaluation components and finds that default settings reverse 9 of 18 pairwise comparisons on AIME '24 relative to the mean across configurations. These papers motivate our work, and they also supply our main competitor: the averaged-over-configurations score treated as ground truth.
What none of them asks is whether a given perturbation family has the same relationship to competence in every model. An ANOVA brittleness term is a pooled quantity, and a factorial mean weights all configurations equally.
FormInv comes closest conceptually. It names prompt sensitivity as differential item functioning, meaning a question's difficulty shifts for reasons unrelated to the skill being tested. It reports striking per-model gaps: Claude Haiku 4.5 flips 41.9% of reworded theorems versus 4.7% for DeepSeek V3. However, it builds standalone flip-rate metrics rather than a latent model, and it never tests whether a perturbation family is non-invariant across models. ProSA and On the Relation between Sensitivity and Accuracy in In-Context Learning tie sensitivity to low decoding confidence. The second paper turns that link into SENSEL, a rule that abstains when answers disagree across prompts. These papers act as a constraint on our design: brittleness concentrates on items the model half-knows, so our measurement model must let format effects depend on item-level competence rather than treating them as constant offsets.
On the SEM side, Aligning MLLM Benchmark With Human Preferences via Structural Equation Modeling supplies the method's spirit. It fits SEM to static task scores, prunes redundant tasks, and builds the GOLD benchmark, which reaches Pearson r = 0.7359 with Chatbot Arena ratings. That work uses one prompt per task and models tasks as indicators across models. We instead model prompt families as indicators within each model and test invariance across models. That is a different question: it asks whether the measuring instrument changes meaning across models, not which tasks to keep.
Activation-level causal mediation work, such as Causal Mediation Analysis for Interpreting Large Language Models, operates inside networks and is not a substitute for behavioral measurement.
Core Claim
For a fixed benchmark and model $g$, let item $i$ have a latent competence $\eta_{ig}$, meaning how well model $g$ actually knows that item. Each perturbation family $j$ yields an ordinal indicator $y_{ijg}$, the number correct out of three random instances of that family. We fit an ordinal confirmatory factor model:
$$y^*_{ijg} = \lambda_{jg}\,\eta_{ig} + \epsilon_{ijg}, \qquad y_{ijg} = k \text{ if } \tau_{jg,k} < y^*_{ijg} \le \tau_{jg,k+1}$$
Here $y^*$ is a continuous propensity to answer correctly, $\lambda_{jg}$ is how sharply family $j$ separates known from unknown items in model $g$, $\tau_{jg,k}$ are thresholds that encode how much harder family $j$ makes items for model $g$, and $\epsilon$ is residual noise.
The invariance hierarchy then becomes a set of testable questions about LLM evaluation:
- Configural invariance: a single competence factor fits for every model.
- Metric invariance: each family discriminates equally across models.
- Scalar invariance: thresholds are equal across models.
A failure of scalar invariance for, say, JSON output in model $g$ is exactly a model-specific format penalty. Under partial invariance, model means of $\eta$ are identified from the anchor families that pass. These means are our proposed competence scores. The claim is falsifiable in two ways. If full scalar invariance holds across models, averaging is already adequate and our method adds nothing. If partial-invariance means do not predict held-out-format rankings better than averages, the latent model is decoration.
Research Program
We will build an evaluation harness with eight perturbation families, drawn from Brittlebench's taxonomy and When Benchmarks are Targets: canonical, padding, typos, LLM paraphrase, persona or emotional framing, option shuffling, rare-symbol labels, and JSON-constrained output. Each family gets three instances per item, and all answers are scored by exact match.
Benchmarks are MMLU, ARC-Challenge, LogiQA and GPQA. FormInv v1 is added because its eight Lean-verified paraphrase families give semantically guaranteed equivalence, which removes the worry that "perturbations" change meaning. The model pool is roughly 25 models, including open-weight Llama 3.1/3.3 and Qwen 3 at several scales, plus a few API models. That yields thousands of items per group, so multi-group ordinal CFA, estimated by weighted least squares, is well powered.
For each benchmark we fit the invariance sequence. We locate non-invariant family-by-model cells using modification indices with a false-discovery correction. We then estimate partial-invariance latent means.
We add one extension, a bifactor variant: a second, format-specific factor loading on the output-structure families (labels, JSON, shuffling). It tests whether "format competence" is a coherent second trait or just family-specific noise. A separate method factor is the alternative a reviewer would propose, so it must be tested rather than assumed.
The same machinery addresses a live contradiction. ProSA reports that few-shot demonstrations stabilize models, while Brittlebench reports that few-shot raises accuracy but worsens brittleness. We will treat zero-shot and five-shot as two groups for the same model. If few-shot raises the latent mean while leaving thresholds invariant, ProSA's reading is right. If it shifts thresholds for format families, Brittlebench's reading is right. If it changes loadings, few-shot changes what the benchmark measures. Each outcome is distinguishable, and neither paper's metric can separate them.
Baselines
The naive reference point is default single-prompt accuracy, the canonical template as a leaderboard would report it. Every paper in this area shows it is unstable, so it sets a floor. When Benchmarks are Targets reports Kendall's τ across perturbations falling to about 0.5.
The strongest existing competitor is mean accuracy over all perturbation configurations, which Beyond Benchmarks explicitly treats as ground truth. This is what a careful practitioner would do today. It is strongest because it already removes random prompt noise and needs no modeling. It will be run on exactly the same items, families and instances as our method, with the same seen/held-out split. To beat it, our scores must achieve a significantly higher rank correlation with held-out-family rankings, tested by bootstrapping over items, and must move less under the planted defect described below. We found no reported result on Lacuna for held-out-format rank agreement of averaged scores, so we will measure this baseline ourselves rather than quote a number.
The second competitor is Brittlebench-style variance decomposition, refit as a crossed random-effects logistic model with item, family and model-by-family terms. This is the strongest statistical rival because model-by-family intercepts are close to scalar non-invariance. It gets the same data, and its model-by-family-adjusted model effects serve as its scores.
We also compare against SENSEL-style consistency filtering, which scores only on items answered consistently across prompts. On GPT-J-6B it improved F1-coverage AUC by 4.1 points over probability baselines, per On the Relation between Sensitivity and Accuracy.
Two ablations isolate the key ingredient. The first is our model with full scalar invariance forced, which amounts to a single latent trait with averaging-like weighting. The second is our model with equal loadings, a Rasch-like constraint. If the partial-invariance model does not beat the forced-invariance ablation, the invariance step is not doing the work.
Why not just…?
Why not just average over many prompt formats? Averaging removes noise that varies randomly by format. It cannot remove a penalty that one model pays on one format, because that penalty survives in the mean in proportion to how often the format appears in the pool. The rankings then depend on which formats someone chose to include. The decisive test is the planted defect below: if averaged scores move and latent means do not, averaging falls short. If the leave-one-family-out experiment shows averaged rankings already as stable as ours, this alternative wins and we will report that.
Why not just fit a mixed-effects model with model-by-format interactions? This is the real threat. Its interaction terms capture most of what scalar non-invariance captures. Two things separate the approaches. Our model has per-model loadings, which allow a format to be uninformative rather than merely harder; this matters given ProSA's finding that brittleness concentrates on low-confidence items. Our model also decides which formats are trustworthy anchors through explicit tests. If the equal-loadings ablation matches the full model and both match the mixed model, we will concede that the contribution is the invariance-testing protocol and its diagnostic findings, not a new estimator.
Why not just report flip rates per model, as FormInv and ProSA do? Flip rates measure how inconsistent a model is. They do not say which formats to discount when comparing competence, and they are confounded with accuracy: FormInv itself finds its inconsistency rate anti-correlated with accuracy. A flip-rate table can show that two models differ in brittleness. It cannot produce a comparable competence score, which is the quantity leaderboards need.
Proposed Experiments
The first decisive check is leave-one-family-out prediction. We fit on seven families and predict the ranking of models under the held-out family, using fresh instances of that family. We measure Kendall's τ and pairwise-reversal rate against the 63% single-perturbation reversal figure from Brittlebench. Partial-invariance means should beat averaged scores on families that pass invariance. For families that fail invariance, the method should flag them as uninformative rather than claim to predict them. The flagging itself is scored by whether flagged families show worse held-out agreement for every baseline.
The second check is a planted format defect with known ground truth. We take one open-weight model and apply a small LoRA fine-tune, a low-cost weight update, that degrades only rare-symbol-label answering. We confirm the defect is format-only by showing unchanged canonical accuracy. A correct method should detect scalar non-invariance for that family and that model and leave the model's latent mean essentially unchanged. Averaged accuracy, by construction, will drop.
The third check is external validity. Following the GOLD paper's use of Chatbot Arena as an anchor, we correlate each scoring method with Arena ratings for the models that have them. We do not claim any number in advance. Finally, we run the zero-shot versus five-shot invariance comparison on MMLU and FormInv to settle the few-shot disagreement.
Success means three things together: better held-out-format stability than mean-over-configurations, robustness to the planted defect, and a clean verdict on the few-shot question. If averaging matches us on the first two, we will report that prompt brittleness in current models is mostly exchangeable noise. That finding would itself be useful, because it would vindicate simple averaging.
References
- Brittlebench: Quantifying LLM robustness via prompt sensitivity
- When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards
- Beyond Benchmarks: Toward Causally Faithful Evaluation of Large Language Models
- FormInv: A Measurement Protocol for Semantic Invariance in Mathematical Reasoning Benchmarks
- ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs
- On the Relation between Sensitivity and Accuracy in In-Context Learning
- Aligning MLLM Benchmark With Human Preferences via Structural Equation Modeling
- Causal Mediation Analysis for Interpreting Large Language Models