Suspicion Mining over Reasoning Traces: Separating Memorized Skills from Missing Ones

When a language model fails a math or logic problem, two very different things may be going on. The model may lack a reasoning skill. Or it may have memorized benchmark items during training, so that its successes on the benchmark were never evidence of the skill at all. Aggregate accuracy blends the two together. Reading thousands of chains of thought (the step-by-step written reasoning a model produces before answering) by hand does not scale. We propose to borrow an old, cheap idea from grammar engineering, statistical error mining, and apply it to reasoning steps. In the original setting, a grammar's parse failures are blamed on the word patterns that recur in failed sentences far more often than in successful ones. Here, instead of words in failed parses, the units are normalized reasoning steps in failed chains of thought.

The key move is what counts as a "failure." Each benchmark problem is paired with a twin: a regenerated version with the same structure but fresh surface details and numbers. We then mine two contrasts. One is failures on both versions, which point to missing skills. The other is success on the original but failure on the twin, which points to memorization-dependent behavior. The output is a ranked tree of step patterns that a developer can inspect. An example leaf might read: "on depth-4 arithmetic, steps that reuse a constant from the original problem's answer." The claim is that this localizes where contamination and genuine gaps live in the reasoning process. Log-probability membership detectors provably cannot do this after reinforcement-learning post-training, and whole-answer judges do not attempt it.

What Already Exists and Where This Differs

Classical error mining provides the method. Error Mining in Parsing Results assigns each word form a suspicion rate through an iterative fix-point over failed sentences, weighted by log frequency. Its best-known result is isolating 18,334 relevant suspicious forms out of over 327,000 on a French corpus. Error Mining with Suspicion Trees turns flat suspicion lists into decision trees whose branches separate distinct failure causes. Fixes guided by these trees cut undergeneration on one surface-realiser input class from 25.3% to 2.1%. Both operate only on discrete symbolic structures such as n-grams, POS tags, and dependency trees, and never on generated reasoning. We reuse their scoring and tree construction. What we add is a new unit (the reasoning step) and a new, twin-based definition of failure.

Four results motivate the proposal.

The closest automated-analysis work constrains the design.

None of the three contrasts original with twin behavior, and none produces step-level, systematic patterns.

Core Claim

Mining step patterns against a twin contrast recovers systematic reasoning defects and memorization-dependent behaviors better than three alternatives: whole-answer judges, embedding clustering, and per-item accuracy gaps. We make this testable by planting both kinds of defect with known ground truth. Detection of contamination per se is not the main claim. The twin accuracy gap already signals contamination in aggregate. What the gap does not tell a developer is which reasoning behaviors carry the inflation and which are genuine gaps masked by it.

Research Program

Twins. Twins come from domains where regeneration is verifiable:

  • DyVal directed-acyclic-graph tasks, where every intermediate node has a known value, so the first incorrect step is labeled exactly;
  • the 3-SAT generator of Can Large Language Models Reason? A Characterization via 3-SAT, with checkable assignments;
  • numeric resamplings of GSM8K and MATH problems whose answers can be recomputed symbolically;
  • ConsisEval easy/hard pairs as a second contrast axis.

The restriction to regenerable domains is a real scope limit, inherited from DyVal. We state it rather than hide it.

Step features. Each chain of thought is split into steps. Each step is mapped to discrete features:

  • normalized step n-grams, with numbers replaced by role tags such as "given," "derived," "unsupported," and "equals-original-answer";
  • operation type;
  • whether the step's value is correct (where verifiable);
  • problem metadata such as DyVal depth or the 3-SAT ratio $\alpha$.

Including difficulty as a feature is deliberate. The Competency Gaps authors note that apparent gaps are often just difficulty confounds. If difficulty is a feature, the tree can split on it first and mine within difficulty strata.

Scoring. Following the fix-point model, for each failed trace $t$ with feature set $F_t$, the blame share of feature $f$ is $$p_{t,f} = \frac{S_f}{\sum_{g \in F_t} S_g},$$ where $S_f$ is the current suspicion of $f$. Suspicion is then re-estimated as the mean of $p_{t,f}$ over all traces containing $f$, and ranking uses $M_f = S_f \ln |O_f|$ with $|O_f|$ the occurrence count. We run this twice, with two definitions of "failed":

  • Gap mining: wrong on both original and twin.
  • Memorization mining: right on original and wrong on twin, restricted to traces from the twin. The original-version success serves as the matched control.

Suspicion trees are then built over the top-ranked features, so that co-occurring causes are grouped rather than listed flatly.

Ground truth by planting. Following the two-stage protocol of the Fragility paper, we contaminate models on a known subset of problems in two ways: supervised fine-tuning (SFT) on benchmark chains of thought, and SFT followed by clean GRPO. Separately, we plant systematic defects by fine-tuning on data containing a specific consistent bug, such as off-by-one when summing over graph children or dropping negations in Boolean tasks. This mirrors Domino's planted-slice evaluation. The models are DeepSeek-R1-Distill-Llama-8B, Llama 3.1 8B, and Qwen 3 4B, all of which appear in the cited work.

Baselines

Twin accuracy gap (naive reference). The model's accuracy on original minus twin, per item and per metadata bucket. This is the "do nothing clever" reference and the method's most direct competitor for contamination. We compute it from exactly the same samples, with the same number of generations per item. Beating it means higher AUROC for identifying contaminated items at the per-item level. It also means that mined patterns explain the gap: removing items that carry the top patterns should shrink the aggregate gap by more than removing an equal number of random high-gap items.

Membership-inference detectors. Min-K%, LiRA, and Loss, run on the same contaminated checkpoints. The Fragility paper reports LiRA at 89.1% AUROC on SFT-only contamination, and 50–58% average AUROC across ten detectors in its Stage II setting. We expect to reproduce these numbers on our planted models as a sanity check. The comparison that matters is after GRPO, where our method must clearly exceed their near-chance AUROC.

Whole-answer error attribution. The MisAttributionLLM approach from Diagnosing Failures in Large Language Models' Answers reports 0.829 Micro-F1 on AttriData error categorization and 0.935 Pearson with human scores, versus 0.817 for GPT-4. We run its released model if available, and a prompted GPT-4-class judge otherwise, on every failed trace. Its per-trace category labels are then aggregated into ranked patterns by frequency. It is the strongest competitor for the "explain failures" question. Beating it means higher precision@10 in recovering planted defects.

Domino-style clustering. An error-aware Gaussian mixture, as in Domino, fitted over step embeddings together with correctness labels, with the same number of output slices as our top-k patterns. It reports 36% planted-slice recovery in its own domains. There is no reported result on Lacuna for any slice-discovery method on reasoning traces.

Ablations. Three ablations isolate the key ingredients:

  • flat suspicion ranking without trees, to test whether hierarchy matters;
  • gap mining alone without the twin contrast, to test whether the twin is what separates memorization from missing skill;
  • raw step n-grams without number-role tags, to test whether the "equals-original-answer" tag is doing all the work.

All methods get the same traces, the same generation budget, and the same k. Any tuned hyperparameters, such as the judge prompt, the number of mixture components, or the tree depth, are chosen on a held-out planting that is excluded from testing.

Why not just…?

Why not just use the twin accuracy gap? This is the real threat for aggregate contamination detection, and we expect the gap to match us there. The gap does not say which behaviors produce the inflation. It is also noisy per item, since each item contributes one binary difference per sample. The decisive experiment is the gap-explanation test in the Baselines section. If removing pattern-carrying items explains no more of the gap than removing random high-gap items, the mining adds nothing for contamination, and we would retreat to the gap-mining (missing-skill) use.

Why not just ask an LLM judge to categorize each failed step? Whole-answer judges already trail specialized models: GPT-4 reaches 0.817 Pearson versus 0.935 for a trained attributor. On intermediate code logic, an LLM judge reaches 73% against 81% for a structural judge (Evaluating Intermediate Reasoning). More fundamentally, a judge labels traces one at a time. It cannot notice that a surface-innocuous step type, such as reusing a particular constant, co-occurs with failure across the corpus, which is what mining measures. The planted-defect comparison settles this directly.

Why not use a step verifier or process reward model? Verifiers such as the one in Training Verifiers to Solve Math Word Problems score candidates per item. On DyVal we already have exact step correctness, so a verifier adds nothing there. Where verifiers help elsewhere, their scores become one more feature for mining rather than a replacement. They say which step is wrong, not which pattern is systematically wrong.

Why not use sparse-autoencoder concept audits? Competency Gaps finds concept-level blind spots cheaply, but its authors acknowledge that activation is not competence. After chain-of-thought contamination, the model's heightened confidence spreads to unseen problems of the same kind, so concept-level scores cannot separate memorized from genuine success. We do not run a full SAE audit. We treat its limitation as motivation for a behavioral twin contrast.

Proposed Experiments

The main experiment measures precision@10 and recall of planted defects and planted contamination patterns. It covers DyVal (arithmetic, Boolean logic, reachability, depths D1–D4), 3-SAT across $\alpha$, and resampled GSM8K/MATH, for each model and each contamination stage.

The first decisive check targets post-GRPO contamination. After clean GRPO, where log-probability detectors are near chance, memorization mining should still surface the planted "reuse original answer or intermediate" behaviors. It should also yield per-item contamination AUROC above both the twin gap alone and the 50–58% detector range.

The second check tests whether memorization and missing skill are actually separated. On models with both planted contamination and a planted bug, memorization mining should rank contamination patterns above the bug, and gap mining should do the reverse. Removing the twin contrast should mix them.

A third check tests transfer on real, unplanted models. We compare the top mined patterns on ConsisEval against its five manually derived failure categories. The measure is how many of those categories the tree recovers without supervision, and how many human-inspection minutes are needed to confirm each leaf versus reading raw failures.

A negative result on the first check would indicate that RL post-training erases behavioral footprints as well as log-probability ones. That would be a substantive finding in its own right, sharpening the Fragility paper's warning.