Do Step-Level Error Detectors Learn the Fact, or the Human's Fingerprint?

Training a model to spot errors in step-by-step reasoning requires examples of reasoning chains with the bad steps marked. Writing such examples from scratch is slow, so the common shortcut is human-in-the-loop synthesis. A language model writes a chain-of-thought (a written sequence of intermediate reasoning steps before the answer), a person finds the faulty step and rewrites it, and the model finishes the chain from that point. Datasets built this way have a known weakness from another area of NLP. People and models who write data under a fixed instruction leave statistical fingerprints, called annotation artifacts. A classifier that sees only part of the input can exploit them. The classic case is a natural language inference model that predicts the label from the hypothesis sentence alone, without reading the premise. We argue that human-edited reasoning chains carry their own fingerprint, the "seam" where a human sentence meets model text. We also argue that the model's continuation after an edit can invent supporting facts to agree with the human. If so, error detectors and process reward models (models that score each reasoning step) trained on such data partly learn to find the seam rather than the falsehood.

This proposal does three things. It measures the seam artifact with a step-only diagnostic, the reasoning-chain counterpart of the hypothesis-only baseline. It measures sycophantic fabrication in continuations that follow an edit. It then tests one construction change against cheaper fixes. In that change, every human correction becomes a minimal pair: the original wrong step and the corrected step are both re-rendered in the model's own voice and placed in the same context. The detector can then separate them only by checking the claim against the question and the evidence. The payoff is concrete. Hallucination detectors trained on cheap edited data should transfer to naturally occurring errors on RAGTruth and ProcessBench, rather than to errors that merely look edited.

Core Claim

Human-in-the-loop chain-of-thought datasets contain a measurable artifact that lets a model locate the faulty or corrected step without the question or any evidence. Detectors trained on such data inherit this shortcut and lose accuracy on naturally occurring hallucinations. Paired re-rendering of each edit removes most of the shortcut and closes most of the transfer gap. Paraphrasing alone does not. Paraphrasing hides the change in style but keeps two other cues. Edited steps tend to contain specific retrieved details such as dates and numbers. Continuations after an edit also carry their own regularities.

Where This Sits

Two systems define the current pipelines. CoTEVer has annotators grade and rewrite GPT-3 reasoning steps alongside retrieved web evidence. On StrategyQA it found that wrong facts (43%) and out-of-date facts (22%) dominate explanation errors. Human-in-the-Loop through Chain-of-Thought selects confused chains by how much sampled answers disagree. A human fixes one sentence, and the model regenerates the rest. That paper reports that a single bad step causes up to half of the failures, and that a fix takes 21 seconds against 49 seconds for solving from scratch. We reuse both protocols as the "author–reviewer" data source. Neither paper checks whether the edits leave a trace that downstream models can exploit, and that check is our contribution.

Three papers motivate the worry. Biases in LLM-Elicited Text rebuilt part of SNLI with LLM writers and found that hypothesis-only BERT reached 86–96% accuracy, against 72% on the human-written original. Machine-written data therefore amplifies artifacts, though that study covered only single sentences and included no human editing. Reward Under Attack perturbed ProcessBench chains and found that a Qwen process reward model barely changed its scores when the question was deleted. That is a "premise-free" behaviour in a trained model. The study looked at the model and never traced the behaviour back to how the training data was built. Language Models Don't Always Say What They Think showed that models rationalize a suggested answer without mentioning the suggestion in 425 of 426 cases, and sometimes invent facts to do so. This motivates our prediction that a human edit acts as a hint, and that the continuation will fabricate in order to support it.

Two papers constrain the design. When Reasoning Traces Become Performative found that both human and LLM judges rate fluent "confabulated continuation" steps as valid. Our sycophancy audit therefore cannot rely on readers judging fluency. It checks each continuation claim against retrieved evidence. CoT Obscures Hallucination Cues showed that chain-of-thought lowers output entropy and degrades uncertainty-based detectors in 465 of 768 configurations. We therefore do not use entropy as the signal for selecting chains. Instead we train a detector that reads the text.

Two papers supply the method and the strongest competitors. Learning to Reason for Hallucination Span Detection trains a 7B model with reinforcement learning to reason before marking hallucinated spans. We use its recipe as the detector we train on each dataset. Adversarial Training for Process Reward Models removes humans entirely: a generator learns to write subtle wrong steps for the reward model to catch. That paper warns that such a loop can co-evolve unnatural cues, a claim our step-only diagnostic can test directly. CoPeD trains students on paired correct and erroneous rationales. It is the closest existing use of contrast, but its pairs come entirely from a teacher model. It never addresses the seams that human editing introduces, and its faithfulness judge is GPT-4, which is susceptible to fluency.

Research Program

We build matched step-labelled corpora in two domains under five construction protocols. The factual domain is multi-hop open-domain questions from StrategyQA, with retrieval as in CoTEVer. The math domain is MATH training problems. Every corpus has the same problems, the same number of labelled steps and the same base generator. Only the construction protocol changes.

The first protocol is MCS-style surgical editing: a human rewrites the faulty step and the model continues. The second is CoTEVer-style evidence-guided revision. The third is fully automated error injection by an adversarially trained generator, following the PRM adversarial-training paper. The fourth applies style normalization: protocol one's chains are paraphrased step by step by the generator model. The fifth is our paired re-rendering.

In paired re-rendering, the model re-renders the human's corrected step under the instruction "restate this claim in your own words in the style of the preceding steps". The original wrong step is re-rendered the same way, and both appear as training examples in identical prefixes. After the edit, the continuation is regenerated twice, once from each version. Continuation steps that add claims unsupported by the retrieved evidence are labelled as errors rather than silently kept. The second regeneration gives a sycophancy measurement for free, because it shows how often the model invents facts to agree with whatever step precedes it.

The analysis has three parts. The first is a step-only diagnostic. A RoBERTa-sized classifier and a 7B model each try to find the labelled step with the question, the evidence and the final answer all removed. We report accuracy against the chance rate, one over the number of steps. The second is a seam probe. Some human edits rewrite steps that were already correct, changing the wording but not the fact. We count how often a trained detector flags these steps. A detector that has learned the fact should leave them alone, and one that has learned the seam will flag them. The third is a sycophancy rate: the fraction of post-edit continuation steps that introduce unsupported claims. Each such claim is checked against retrieval by two annotators.

Baselines

The naive reference point is a detector trained on raw MCS-style data with no artifact treatment, together with the untrained Qwen2.5-7B base model. The file reports the base model at 16.2 Span-F1 on RAGTruth, from the hallucination span detection paper.

The strongest existing method for factual span detection is RL4HS from Learning to Reason for Hallucination Span Detection. It reports 55.9 Span-F1 on RAGTruth, against 50.1 for supervised fine-tuning and 51.2 for o3. It is the strongest competitor because it already reasons step by step and was trained on naturally occurring errors. We run its recipe unchanged, with the same 7B base, the same reward and the same number of training steps, on each of our five corpora and on RAGTruth's own training split. Each corpus is matched in token count, and every arm gets the same hyperparameter search budget. For our data to count as better, a detector trained on paired re-rendered data must reach or exceed 50.1 Span-F1 on RAGTruth, the supervised result on in-domain data, without seeing RAGTruth training data. It must also beat the unedited MCS corpus by a margin outside seed variance across three seeds.

On the math side, the reference points are ProcessBench's reported F1 scores: 61.9 for GPT-4o, 71.5 for QwQ-32B-Preview and 87.9 for o1-mini, from ProcessBench. The file gives no reported ProcessBench result on Lacuna for the adversarially trained reward model. We therefore retrain that method ourselves on matched problems and compute so it can serve as the automated competitor.

The ablation that isolates the key ingredient compares style normalization alone (protocol four) with paired re-rendering (protocol five). The two differ only in whether a matched wrong-step counterpart exists and whether the continuation is regenerated under the evidence check. If protocol four matches protocol five on both the step-only diagnostic and transfer, the pairing is unnecessary and we will say so.

Why not just…?

Why not just train on naturally occurring errors like RAGTruth or ProcessBench? These corpora are the target distribution, which is why we use them only for evaluation. They are expensive to build. ProcessBench discarded 30% of its initial solutions because doctoral-level annotators could not agree, and RAGTruth needed span annotation of nearly 18,000 responses. The question here is whether far cheaper edited data can substitute for them. The RAGTruth-trained arm gives the upper bound that our protocols are measured against.

Why not just paraphrase every step so the human's style disappears? This is the most serious threat, and it is protocol four. We expect it to fail for two reasons. Human corrections carry content cues, such as specific dates and figures taken from retrieved evidence, that survive paraphrase. The continuation after an edit also has its own sycophantic regularities, which paraphrase leaves in place. If the step-only diagnostic and the RAGTruth transfer score show no gap between protocols four and five, paraphrase is the answer. The contribution then shrinks to the diagnostic and the sycophancy measurement.

Why not just remove humans and use adversarial error generation? Adversarial Training for Process Reward Models reports gains of 3.4 points over ReST-MCTS, and 5.3 points out of distribution on SciBench. Its authors note, though, that the generator struggles to produce certain natural error types. Biases in LLM-Elicited Text also suggests that fully machine-written data carries the strongest artifacts. Protocol three is measured with the same step-only diagnostic. If its artifact level is low and it transfers as well as protocol five, the human edit adds nothing beyond cost.

Why not just debias by filtering or reweighting the examples the step-only classifier solves? The file records that removing known cues often makes models find other shortcuts. We include this as an extra arm: protocol one with examples the step-only classifier solves filtered out. We retrain the diagnostic on the filtered data to check whether new cues appear.

Proposed Experiments

The first decisive check is whether the artifact exists at all. On the MCS-style corpus, the step-only localizer should beat chance by a wide margin, while on paired re-rendered data it should fall to near chance. If the raw edited data already sits near chance, the central premise fails.

The second check asks whether the artifact predicts transfer. Across the five protocols, the two domains and three seeds, we regress RAGTruth Span-F1 and ProcessBench F1 on step-only accuracy on the training data. A clear negative slope supports the claim, and a flat one refutes it.

The third check, the seam probe, should show detectors trained on raw edited data flagging correct but reworded steps at a far higher rate than detectors trained on paired data.

The fourth check counts sycophantic fabrications. We report the fraction of post-edit continuation steps with unsupported claims on StrategyQA, and how much that fraction falls when continuations are regenerated under the evidence check. The file has no reported result on Lacuna for this quantity.

The fifth check repeats the premise-free test from Reward Under Attack on our trained math reward models. We delete the question and measure the score change, expecting near-zero sensitivity for models trained on raw edited data and strong sensitivity for those trained on paired data. That result would tie the premise-free behaviour found there to how the training data was built.

Finally, HaluEval QA serves as a secondary transfer target. The file reports ChatGPT at 62.59% detection accuracy there, from HaluEval.