Agreeing on the Answer but Not the Reason: Measurement Limits of LLM Reasoning Failure Taxonomies

Madhu Sudhan Rao BeesettyarXiv 2026 · arXiv

TL;DR

Don’t take LLM failure taxonomies at face value. This paper demonstrates that reported rates of specific reasoning errors—like "Compositional Collapse"—are often artifacts of the measurement procedure (such as prompt wording and classification order) rather than stable properties of the models themselves.

What Problem Does This Paper Address?

As Large Language Models (LLMs) become more capable, researchers are moving beyond simple accuracy scores to categorize how they fail. Taxonomies like "Reasoning Drift" or "Compositional Collapse" are increasingly used to compare models. The "LLM-as-a-judge" pattern is the standard way to apply these taxonomies at scale.

However, this paper asks a critical question: Are these failure categories actually reproducible? If two different judges (or one judge with a slightly different prompt) look at the same error, will they agree on why it happened? The authors find that while judges agree perfectly on whether an answer is right or wrong, they diverge wildly when assigning a "reason" for the failure.

Methods

The authors focused on a specific failure mode they call Compositional Collapse: when a model explicitly notices a required part of a problem (e.g., "there are 13 spotted puppies") but then ignores or misapplies that information in a later step.

Their methodology involved:

  1. Generation: 12 different models (proprietary and open-weight) solved math problems from a multilingual benchmark across 11 languages.
  2. Inspection: A fixed "inspector" model (Claude 3.5 Sonnet) categorized the failures using a structured JSON prompt.
  3. Human Validation: Native speakers of all 11 languages re-rated a subset of 179 transcripts to see if they agreed with the inspector.
  4. Sensitivity Analysis: The authors permuted the order in which the classifier tested for different failure modes to see if "tie-breaking" influenced the results.
  5. Prompt Engineering: They re-ran the inspector using the exact, simpler instructions given to the human raters to isolate the effect of the prompt.

Datasets

The study used mGSM-Symbolic, a multilingual, symbolic variant of the popular GSM8K math benchmark. It includes 22 problem templates translated into 11 languages (including Bengali, Chinese, Swahili, and Thai), resulting in 2,316 graded records for analysis.

Key Insights and Ideas

  • The "First-Match" Fallacy: Many failures (over 50%) carry evidence for multiple categories. If your classifier checks for "Collapse" before "Drift," it will label the item as "Collapse." Swapping this order flipped the "dominant" failure mode in 11 out of 12 models.
  • Agreement Collapses on the "Why": Humans and models agree almost perfectly on whether an answer is correct ($\kappa \approx 0.9$). But on the reason for failure, agreement was near chance ($\kappa \approx 0.09$).
  • It’s the Prompt, Not the Model: When the inspector model was given the same simple instructions as the humans, its agreement with humans jumped from $\kappa = 0.147$ to $\kappa = 0.636$. The "failure" was in the instrument's design, not the judge's reasoning capability.
  • Benchmark "Laundering": The authors found that some "reasoning failures" were actually caused by bad translations in the benchmark (e.g., a crucial clause missing in the Bengali version). The inspector model "laundered" these benchmark defects into "model failure" statistics.

Key Results

The sensitivity of the taxonomy to classification order is starkly illustrated in the authors' results:

Table I: The same classifier under two different test orderings. Note how the "Plurality" category flips for almost every model.

By simply changing the order of tests, the reported share of "Compositional Collapse" failures could swing by as much as 50 percentage points.

Furthermore, the gap between human and model judges was massive until the prompt was aligned. As shown in the comparison below, two independent human rounds agreed with each other reasonably well ($\kappa = 0.659$), but the original inspector was an outlier:

Table III: Agreement on the Compositional Collapse label. The "H-style insp." (human-style prompt) significantly closes the gap with human raters.

Limitations and Future Work

  • Sample Size: The human validation set was relatively small (179 items), and for some languages, only a few items were reviewed.
  • Domain Specificity: The findings are limited to grade-school math word problems. It remains to be seen if these measurement limits apply to other reasoning domains like coding or legal analysis.
  • Self-Inspection Bias: The inspector model (Claude 3.5 Sonnet) was also one of the models being tested. The authors noted that it was the only model whose results remained stable under tie-break permutations, suggesting a potential "same-family" bias.
  • Future Work: The authors recommend four "cheap checks" for future research: reporting label co-occurrence, permuting tie-breaks, validating on independent human samples, and auditing systematic disagreements.

Summary updated

Abstract

Failure taxonomies for LLM reasoning are increasingly reported as measurements: papers state what fraction of a model's errors fall into named categories, then compare those fractions across models. We show that for one such category the reported fraction is largely a property of the measurement procedure. We defined Compositional Collapse (a model notices a required sub-part of a problem, then drops or misapplies it) and measured it across twelve models and eleven languages on a public multilingual math benchmark (2,316 graded records). First, 50.1% of failures carry evidence for more than one category, so the reported plurality depends on the order in which the classifier tests them: reversing two tests moves it in eleven of twelve models, and across all six orderings Collapse's share spans 26.4% to 76.4%. Second, agreement collapses for the category but not for the task. Native speakers of all eleven languages judged 179 transcripts, agreeing with our inspector at kappa = 0.932 on answer correctness but only kappa = 0.093 on Collapse. A second native speaker per language re-rated 142 items; on 121 determinate pairs the two humans agree with each other at kappa = 0.659, while each agrees with the inspector at kappa <= 0.094. Re-running the same inspector model with the human instructions verbatim raises kappa against Human 2 from 0.147 to 0.636, so the gap is largely attributable to the prompt, not the model. Third, the inspector labels Collapse 1.81 times as often as native speakers, but only 1.17 times once a defective benchmark item (a deleted clause in the Bengali and Chinese translations) is excluded. We recommend four cheap checks before comparing failure-mode rates across models: report label co-occurrence, permute the tie-break, validate on a shared independent sample, and audit systematic disagreements.
BibTeX
@misc{beesetty2026agreeing,
  author = {Madhu Sudhan Rao Beesetty},
  title = {{Agreeing on the Answer but Not the Reason: Measurement Limits of LLM Reasoning Failure Taxonomies}},
  year = {2026},
  eprint = {2611.13140},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.13140},
}
Figure