TL;DR
Don’t take LLM failure taxonomies at face value. This paper demonstrates that reported rates of specific reasoning errors—like "Compositional Collapse"—are often artifacts of the measurement procedure (such as prompt wording and classification order) rather than stable properties of the models themselves.
What Problem Does This Paper Address?
As Large Language Models (LLMs) become more capable, researchers are moving beyond simple accuracy scores to categorize how they fail. Taxonomies like "Reasoning Drift" or "Compositional Collapse" are increasingly used to compare models. The "LLM-as-a-judge" pattern is the standard way to apply these taxonomies at scale.
However, this paper asks a critical question: Are these failure categories actually reproducible? If two different judges (or one judge with a slightly different prompt) look at the same error, will they agree on why it happened? The authors find that while judges agree perfectly on whether an answer is right or wrong, they diverge wildly when assigning a "reason" for the failure.
Methods
The authors focused on a specific failure mode they call Compositional Collapse: when a model explicitly notices a required part of a problem (e.g., "there are 13 spotted puppies") but then ignores or misapplies that information in a later step.
Their methodology involved:
- Generation: 12 different models (proprietary and open-weight) solved math problems from a multilingual benchmark across 11 languages.
- Inspection: A fixed "inspector" model (Claude 3.5 Sonnet) categorized the failures using a structured JSON prompt.
- Human Validation: Native speakers of all 11 languages re-rated a subset of 179 transcripts to see if they agreed with the inspector.
- Sensitivity Analysis: The authors permuted the order in which the classifier tested for different failure modes to see if "tie-breaking" influenced the results.
- Prompt Engineering: They re-ran the inspector using the exact, simpler instructions given to the human raters to isolate the effect of the prompt.
Datasets
The study used mGSM-Symbolic, a multilingual, symbolic variant of the popular GSM8K math benchmark. It includes 22 problem templates translated into 11 languages (including Bengali, Chinese, Swahili, and Thai), resulting in 2,316 graded records for analysis.
Key Insights and Ideas
- The "First-Match" Fallacy: Many failures (over 50%) carry evidence for multiple categories. If your classifier checks for "Collapse" before "Drift," it will label the item as "Collapse." Swapping this order flipped the "dominant" failure mode in 11 out of 12 models.
- Agreement Collapses on the "Why": Humans and models agree almost perfectly on whether an answer is correct ($\kappa \approx 0.9$). But on the reason for failure, agreement was near chance ($\kappa \approx 0.09$).
- It’s the Prompt, Not the Model: When the inspector model was given the same simple instructions as the humans, its agreement with humans jumped from $\kappa = 0.147$ to $\kappa = 0.636$. The "failure" was in the instrument's design, not the judge's reasoning capability.
- Benchmark "Laundering": The authors found that some "reasoning failures" were actually caused by bad translations in the benchmark (e.g., a crucial clause missing in the Bengali version). The inspector model "laundered" these benchmark defects into "model failure" statistics.
Key Results
The sensitivity of the taxonomy to classification order is starkly illustrated in the authors' results:
By simply changing the order of tests, the reported share of "Compositional Collapse" failures could swing by as much as 50 percentage points.
Furthermore, the gap between human and model judges was massive until the prompt was aligned. As shown in the comparison below, two independent human rounds agreed with each other reasonably well ($\kappa = 0.659$), but the original inspector was an outlier:
Limitations and Future Work
- Sample Size: The human validation set was relatively small (179 items), and for some languages, only a few items were reviewed.
- Domain Specificity: The findings are limited to grade-school math word problems. It remains to be seen if these measurement limits apply to other reasoning domains like coding or legal analysis.
- Self-Inspection Bias: The inspector model (Claude 3.5 Sonnet) was also one of the models being tested. The authors noted that it was the only model whose results remained stable under tie-break permutations, suggesting a potential "same-family" bias.
- Future Work: The authors recommend four "cheap checks" for future research: reporting label co-occurrence, permuting tie-breaks, validating on independent human samples, and auditing systematic disagreements.
Summary updated