Many claims about where large language models lack professional knowledge rest on very small test sets. Legal entailment results are reported on 81 or 109 bar-exam questions. Biomedical question answering is scored in small batches. Frontier math is judged on 30 contest problems. On sets this small, rewording the instructions, changing the random seed, or drawing a different handful of items can move accuracy by several points. That is often as large as the effect being claimed, for example "theorem hints do not help," "legal-writing prompts help," or "reasoning models hit a wall on appellate reversal." This proposal builds and validates an evaluation protocol that answers a narrow question for such claims: is the measured difference between two conditions larger than what item sampling, prompt wording, and decoding randomness would produce by themselves? We then use that protocol to re-audit specific published domain-gap findings.
The central idea is to borrow ground truth from the few domain benchmarks that happen to be large. AppealCase has 10,000 matched civil judgments, and TheoremQA has 800 theorem-driven questions. On these sets, the "true" difference between two conditions is known to high precision. We can therefore repeatedly draw small subsets of size 30, 81, 109, or 200 and count how often each small-sample protocol gets the sign wrong or produces a confidence interval that misses the full-set answer. Protocols that survive this test are then applied to the benchmarks that are small and cannot be enlarged.
Core Claim
Our hypothesis is that the dominant hidden error in small expert benchmarks is not item sampling but prompt-wording variance. Standard paired tests ignore it, because they fix one prompt. Continuous scores do not remove it either, because a rewording shifts the model's behavior systematically across all items rather than adding independent noise per item. To make this precise, let $\hat\Delta$ be the estimated accuracy difference between two conditions, averaged over $N$ items, $P$ prompt paraphrases, and $S$ decoding seeds. Treating these three as crossed random factors gives
$$ \mathrm{Var}(\hat\Delta) \approx \frac{\sigma^2_{\text{item}}}{N} + \frac{\sigma^2_{\text{prompt}}}{P} + \frac{\sigma^2_{\text{seed}}}{NPS}, $$
where $\sigma^2_{\text{item}}$ is the variance of the per-item difference, $\sigma^2_{\text{prompt}}$ is the variance of the condition effect across paraphrases, and $\sigma^2_{\text{seed}}$ is the residual decoding variance. This form follows from averaging independent random effects. It shows the key point: when $P=1$, collecting more items never shrinks the prompt term.
The testable claims are threefold. First, confidence intervals that set $P=1$ will under-cover the full-set truth on AppealCase and TheoremQA subsamples, even when those intervals come from a paired bootstrap. Second, a protocol that samples paraphrases and uses a continuous per-item score will reach nominal coverage at $N\approx100$. Third, under that protocol, some published domain-gap conclusions will no longer be distinguishable from noise, while others will hold up. Naming which ones hold is itself the deliverable.
Relation to Prior Work
The motivation comes from two sets of findings. On the variance side, Quantifying Variance in Evaluation Benchmarks trained ten Llama-2-7B seeds and found three things: 1–2% swings from seed alone, continuous metrics with 10–100 times higher signal-to-noise ratio than accuracy, and wrong pairwise comparisons in roughly 12% of cases for Item Response Theory "tiny benchmarks." Its setting was general benchmarks during pretraining, not expert domains where small sets are unavoidable. On the prompt side, Exploring the Effectiveness of Prompt Engineering for Legal Reasoning Tasks reported nearly 4% accuracy swings from minor wording changes on COLIEE, with $N\le109$. That paper documents the problem but offers no estimator. Biomedical Question Answering with Transformer Ensembles likewise reports large variance across BioASQ test batches without modeling it.
The claims we audit come from the domain-gap literature. TheoremQA concluded that supplying theorem descriptions does not substantially help. AppealCase found that all 20 models score below 50% F1 on first-instance reversal prediction and that DeepSeek-R1 barely beats standard models. The prompt-engineering paper reports IRAC/TREACC gains over competition winners, with p = 0.0138 on COLIEE 2022. Context Variance Evaluation of Pretrained Language Models for Prompt-based Biomedical Knowledge Probing is the closest precedent for the view that apparent knowledge deficits can be prompt artifacts: ClinicalBERT's weakness on BioLAMA disappeared under context perturbation. That work, however, probes masked models on single-token facts. We apply the same view to generative models on multi-step expert tasks and attach calibrated uncertainty to it.
The closest methodological work is Evaluating LLMs When They Do Not Know the Answer. It builds a semiparametric one-step estimator that uses a model's pairwise judgments of other models' solutions as control variates. This reduces DeepSeek-V3.2's naive estimation error on AIME 2025 from 3.33% to under 1%. We differ from it in three ways. It estimates one model's accuracy on a single fixed prompt, whereas we estimate differences between conditions and treat wording as a random factor. It depends on comparative-verification signals, which it acknowledges are weak when a model fails at both generating and verifying answers. That is plausibly the situation in appellate law, where every model is below 50% F1. Finally, it was validated only on math and graduate science.
Two further lines of work constrain our design. Measuring Nines of Reliability needs programmatic templates, which legal and clinical items lack, so we do not use importance sampling. Meta simulation approach for evaluating machine learning method selection in data limited settings reduces variance by simulating from a learned causal graph. It works only on tabular data, and it warns that simulation fidelity does not guarantee correct rankings, so we avoid synthetic items altogether.
Research Program
We will build a crossed evaluation harness with three components. The first is a set of paraphrases: for each task, 8–12 instruction wordings, written by the authors and filtered for meaning-preservation by a second annotator. The second is several decoding seeds per wording. The third is a per-item continuous score, computed as the probability mass on the correct label for open-weight models, or as the correct-answer frequency over sampled completions for closed models. The estimator fits a crossed random-effects model to the per-item paired differences between conditions. From this fit it reports the condition effect, an interval that accounts for all three variance components, and each component's share of the total variance.
The validation stage uses AppealCase first-instance reversal prediction and TheoremQA. On each, we define 4–6 condition contrasts that mirror published claims, such as with versus without theorem augmentation, chain-of-thought versus a reasoning model, and structured legal prompt versus plain prompt. We evaluate every contrast on the full set to get the reference effect. We then draw 2,000 random subsets at each of $N\in\{30,81,109,200\}$ and run every protocol on every subset.
The audit stage applies the validated protocol to COLIEE 2021 and 2022, the BioASQ 11b yes/no questions, and AIME-scale subsets. These are the benchmarks where the published claims cannot be checked by simply adding data. Because text-davinci-003 is no longer available, we re-run each claim with current open and closed models, and we report whether the claimed direction and size replicate. We will not assert anything about the original runs themselves.
Baselines
The naive reference point is single-prompt, single-seed accuracy with a paired bootstrap over items. This is what most domain papers report, and it is the protocol behind the COLIEE claims: 80.25% zero-shot TREACC on the 2021 set of 81 questions versus 70.37% for the prior winner. Every protocol sees identical items and model outputs, so the naive baseline gets its number from the first paraphrase and first seed of the same runs. On subsampled sets, we count the proposal as beating this baseline if its interval coverage of the full-set effect is closer to 95% at equal $N$ and its sign-error rate is lower. No paired-bootstrap coverage figure is reported on Lacuna for these domain benchmarks. For AppealCase, the only reported number is below 50% F1 for all 20 models.
The strongest existing method is the comparative-signal one-step estimator from Evaluating LLMs When They Do Not Know the Answer. Its reported result is a reduction of naive accuracy-estimation error for DeepSeek-V3.2 on AIME 2025 from 3.33% to under 1%. We run it on the same subsets, using the same pool of candidate models to generate comparison solutions and the same budget of model calls. The call budget is matched by giving it the calls our protocol spends on paraphrases and seeds. It estimates per-condition accuracy, and we take the difference between conditions. Beating it means better coverage of the full-set difference at equal inference cost on AppealCase and TheoremQA. We expect it to stay competitive on AIME, where comparative signals are strong.
Two ablations isolate our ingredients. The first keeps the continuous score but fixes $P=1$; this is the approach of Madaan et al. applied to domain contrasts. The second keeps paraphrase sampling but uses 0/1 accuracy. If the first ablation already matches the full protocol's coverage, the prompt-variance hypothesis is wrong. We would then report that continuous scoring is sufficient, and the audit would proceed with that simpler protocol.
Why not just…?
Why not just use a paired bootstrap on the existing set? A bootstrap resamples items, so it measures only the first term of the variance decomposition. The COLIEE wording swing of nearly 4% is invisible to a bootstrap run on a single prompt. The subsampling experiment directly measures how badly bootstrap intervals under-cover when the prompt term is large. If they turn out not to under-cover, that is a clean negative result, and the bootstrap is vindicated.
Why not just switch to continuous metrics, as Madaan et al. recommend? This is the most serious threat to the proposal. Continuous scores reduce per-item noise dramatically according to Quantifying Variance in Evaluation Benchmarks. Our argument that they leave the systematic prompt shift untouched is a hypothesis, not a known fact. The first ablation settles it. If it wins, the contribution narrows to the audit plus the demonstration that continuous scoring suffices in expert domains, which is still useful.
Why not just apply the comparative-signal estimator to law and medicine? That estimator assumes the model can verify solutions it cannot produce. Its authors flag that the method degrades when a model fails at both, and the uniformly low AppealCase scores suggest exactly that regime. The method also fixes the prompt. We run it head-to-head rather than dismissing it.
Why not just curate a compact, discriminative subset with Item Response Theory, or collect more expert labels? Madaan et al. found that IRT-selected subsets increase seed variance and flip about 12% of pairwise comparisons. Collecting more labels is ruled out by the cost of expert annotation, which is the reason these benchmarks are small in the first place.
Decisive Checks
The main result is a curve of coverage and sign-error rate against $N$ for each protocol, on AppealCase and TheoremQA, with the estimated share of variance attributable to prompt wording reported for each contrast. The proposal succeeds if two conditions hold: the prompt share is substantial, meaning comparable to the item share at $N\approx100$, and only protocols that sample paraphrases reach nominal coverage. The audit succeeds if it produces a clear verdict for each claim. For instance, if the theorem-augmentation null holds with a tight interval around zero, it is a real knowledge-application deficit. If the IRAC gain on COLIEE shrinks inside paraphrase variance, it is not established. Either outcome changes how the domain-gap literature should read its own small benchmarks.
References
- Quantifying Variance in Evaluation Benchmarks
- Exploring the Effectiveness of Prompt Engineering for Legal Reasoning Tasks
- Biomedical Question Answering with Transformer Ensembles
- TheoremQA: A Theorem-driven Question Answering Dataset
- AppealCase: A Dataset and Benchmark for Civil Case Appeal Scenarios
- Context Variance Evaluation of Pretrained Language Models for Prompt-based Biomedical Knowledge Probing
- Evaluating LLMs When They Do Not Know the Answer: Statistical Evaluation of Mathematical Reasoning via Comparative Signals
- Measuring Nines of Reliability: Sample-Efficient LLM Evaluation on Saturated GSM Benchmark
- Meta simulation approach for evaluating machine learning method selection in data limited settings