When the Error Can't Be Found: Verified Failure Attribution as a Detector of Benchmark Label Errors

When a language model's answer disagrees with a benchmark's "gold" label, evaluation pipelines assume the model is wrong. Error-analysis tools then sort that disagreement into a category such as calculation slip, misread premise, or missing fact. That assumption is now often false. On SVAMP and SQuAD2.0, more than 80% of recorded frontier-model errors turned out to be benchmark flaws, and once those were removed, real systematic failures became visible (Do Large Language Model Benchmarks Test Reliability?). The two research threads that should catch this, sorting model failures into types and cleaning noisy benchmark labels, have mostly ignored each other. As a result, an error classifier handed a mislabeled item will make up a plausible reason for a "mistake" that never happened.

This proposal turns that weakness into a measurement. If a model's answer is genuinely wrong, a specific faulty step exists that can be pointed to and checked: an arithmetic operation that does not evaluate as written, a quantity copied wrongly from the question, a unit mismatch. If the model is right and the label is wrong, any claimed fault has to be fabricated, and a fabricated fault will fail an independent check. We ask an attributor model to name a concrete, checkable fault for each disagreement. We then verify that claim with deterministic tools wherever possible, recomputing arithmetic or matching spans against the question. Items for which no claimed fault survives verification are flagged as likely label errors. The same procedure yields an error-type breakdown with label errors removed, which no existing taxonomy provides.

Core Claim

Whether a precise, verified failure explanation exists is an informative signal for separating label noise from genuine model error. The hypothesis to test is that this signal is partly independent of the two signals in use today, agreement across models and self-consistency across samples, and that the independence is greatest where those two signals fail: items on which many models make the same mistake. The direct test is whether adding the verification signal improves precision at matched recall on expert-adjudicated benchmarks, measured against both those signals and against simply asking a strong model whether the label is wrong. If it does not, the idea fails.

How This Relates to Prior Work

Three papers motivate the gap. The CoNLL-2003 study Named Entity Recognition - Is There a Glass Ceiling? already put "dataset errors" as one tier of a model-error taxonomy and re-annotated the test set by hand. That makes it the closest precedent for a taxonomy covering both model errors and label errors. Its separation was manual and limited to token labeling, and it found that 27.8% of errors were shared by all four architectures, which is exactly the case where agreement-based cleaning breaks down. Platinum Benchmarks (Do Large Language Model Benchmarks Test Reliability?) flags any item where some frontier model disagrees with gold and sends it to experts. That flag is high-recall but filters nothing on its own, and it cannot see shared mistakes. Beyond the Singular samples 50 generations and flags items the model answers consistently yet is always scored wrong. On GSM8K those flags were confirmed label errors 44.4% of the time, so the other 55.6% were confident genuine errors that consistency alone cannot separate from label errors. Our method targets exactly that remainder.

Two papers constrain the design. Error Taxonomy-Guided Prompt Optimization clusters failed traces into error categories and gains 6–7 points on FOLIO. It treats every failure as real, which makes it a useful downstream victim: noisy validation labels should inject fabricated categories into its guidance. Scaling Generative Verifiers shows that model judges reward surface style, that strong models accept invalid proofs, and that heuristics with no mathematical knowledge reach up to 75% on proof benchmarks. This is why our verification step relies on deterministic checks rather than another model's opinion wherever the domain permits. It is also why we restrict the main study to domains where faults can be checked mechanically.

The method borrows step-level fault localization from AgentHallu and the arithmetic, logical and contextual error types from Can LLMs Learn from Previous Mistakes?. AgentHallu also supplies a warning: localization is hard, with Gemini-2.5-Pro at 41.1% step accuracy. A weak attributor that fails to find real faults would produce false label-error flags, and the experiments are designed to measure exactly this.

Research Program

The pipeline has three stages. First, we take each item where the model's answer disagrees with gold and prompt an attributor, under a fixed error taxonomy, to produce a fault claim. A fault claim has three parts: the step index, the error type, and a machine-checkable assertion such as "step 3 states 17×4=78" or "the question gives 12 apples; step 1 uses 21." Second, a verifier checks each assertion. Arithmetic is re-executed, quoted quantities are matched against the question text, and unit and rounding claims are recomputed. Claim types that cannot be checked mechanically are logged separately and never counted as verified. Third, we sample $m$ independent fault claims per item and combine them into a noise score:

$$s(x) = c(x)\,\bigl(1 - v(x)\bigr),$$

Here $c(x)$ is the semantic consistency of the model's own answers across samples, as defined in Beyond the Singular, and $v(x)$ is the fraction of the $m$ fault claims that pass verification. A high score means the model answers confidently and no one can show where it went wrong. Items with low scores receive the type of their verified fault, which gives the noise-corrected taxonomy distribution.

The analysis has three parts. First, we measure how often the attributor fabricates faults on items known to be correct. Second, we measure how often it misses faults on items known to be wrong. Third, we measure how each error category's share changes once label errors are removed. The prediction for the third part is that "context" and "linguistic" categories shrink most, because a mislabeled item is most easily explained away as the model misreading the question.

Baselines

The naive reference point flags every model–gold disagreement as a label error. This baseline matters more than it sounds. On SVAMP and SQuAD2.0 it would already be correct more than 80% of the time, given the Platinum finding, so we report precision relative to each benchmark's base rate and treat GSM8K, where the base rate is lower, as the more informative test.

The agreement baseline is the Platinum filter: an item is flagged if any of a fixed panel of models disagrees with gold. We also run a stricter variant that requires all panel models to disagree, so that it can be ranked by agreement count. The consistency baseline is the Data Maps quadrant from Beyond the Singular, whose reported figure is 44.4% precision on GSM8K among high-consistency, zero-accuracy items. That is the number to beat on GSM8K at the same flagged-set size.

The strongest competitor is direct adjudication. A strong model sees the question, the gold label and the model's answer and is asked which is correct, given the same token budget as our full pipeline. It is the cheapest way to get a decision that reads the actual reasoning.

Two ablations isolate the key ingredient. The first, unverified attribution, uses the attributor's own confidence that it found a fault, with deterministic checks removed. The second keeps verification but drops the taxonomy, asking only "is there any error?" without a typed, checkable assertion.

All methods use the same model pool, the same items, the same 50 samples where sampling applies, and an equal compute budget for adjudication. Thresholds are tuned on a held-out split. Beating a baseline means higher precision at matched recall, measured by area under the precision–recall curve, against expert-adjudicated labels. The required margin is one that holds across seeds and holds specifically on the shared-error subset.

Why not just…?

Why not just ask a strong model whether the gold label is wrong? This is the real threat. If direct adjudication matches our method, the verification machinery is unjustified and we will say so. The reason to expect a gap comes from Scaling Generative Verifiers: judges accept flawed reasoning that looks fluent and share blind spots with the models they judge. That failure should be concentrated on items where models make the same mistake. The head-to-head comparison on the shared-error subset settles this question. Our method should win there, or the claim fails.

Why not just use sampling consistency, as in Data Maps? Consistency detects confidence, not correctness. Its reported 44.4% precision on GSM8K means most confident disagreements are real errors that it cannot tell apart from label errors. Our score multiplies consistency by verification failure, and the ablation that sets $v(x)=1$ recovers the Data Maps baseline exactly, so any gain is attributable to verification.

Why not just harden the benchmark, MMLU-Pro style? MMLU-Pro widens score margins (GPT-4o 72.6%) and reduces sensitivity to prompt wording, but it relies on LLM-generated distractors that can themselves be valid answers. Hardening also does nothing to explain individual failures or to clean existing benchmarks. It is a complement, not a substitute. As a secondary test we will run our detector on MMLU-Pro items with computable answers to look for exactly those distractor errors.

Proposed Experiments

The primary evaluation uses the expert-adjudicated GSM8K and SVAMP items from Platinum Benchmarks as ground truth for which disagreements are label errors. Models are an open model family at two scales plus one frontier API model, with a separate attributor model. Metrics are precision–recall curves for label-error flags, the fabrication rate on items whose gold was verified correct, and the miss rate on items whose gold was verified wrong.

The second evaluation is controlled corruption. Starting from Platinum-cleaned items, we perturb gold answers at rates of 5%, 15% and 30%, choosing perturbations that are plausible (off-by-one, rounding, unit errors) rather than random. This gives known noise at scale and tests whether detection degrades smoothly.

The third evaluation targets the shared-error regime. We take the subset of items on which every model in the pool gives the same wrong answer and report all methods on it separately. This is where the core claim lives.

The decisive downstream check reruns ETGPO-style taxonomy guidance on FOLIO and MMLU-Pro validation sets with injected label noise, comparing guidance built from all failures against guidance built from failures our score retains. We expect the reported 6–7 point FOLIO gain to erode under noise and to be partly recovered by filtering. We also expect the noise-corrected guidance to contain fewer error categories that the clean set never produces.

As an exploratory extension, we will apply the same logic of "no verifiable fault, therefore suspect the label" to MedCalc-Bench, where calculation and extraction errors can be checked. No reported result was found on Lacuna for MedCalc-Bench baselines, so this extension will establish its own reference numbers.

The idea fails if verification adds nothing over direct adjudication on the shared-error subset, or if the fabrication rate is so high that verified faults appear just as often on correct answers. Either outcome is informative, because it would show that models can produce checkable but false fault claims, a failure mode that error taxonomies currently do not measure.