Finding the First Wrong Step: Calibrated Debate for Mechanistic Reasoning Chains

When a language model reasons step by step, one wrong early step is usually enough to sink the whole answer. Later steps treat the mistake as settled fact and build consistently on top of it. Models are also bad at noticing where they went wrong. On a benchmark of human-annotated reasoning traces, GPT-4 located the first erroneous step only 52.87% of the time, yet corrected traces well once someone told it where the error was (LLMs cannot find reasoning errors, but can correct them given the error location). The missing piece is therefore a reliable pointer to the first wrong step. A pointer is only useful if its confidence can be trusted. An overconfident pointer that says "the chain is fine up to step 7" when step 2 is broken is worse than no pointer at all.

We propose to use multi-agent debate, in which several model instances argue for and against a claim, only for that narrow job. Debate will not be asked to produce final answers. It will be asked to score each step of a reasoning chain for validity, and those scores will be passed through a conformal calibration procedure. Conformal calibration is a statistical method that sets a decision threshold on held-out data so that a stated error rate is guaranteed. The output is a certified prefix: the longest opening segment of the chain that we can claim, with probability at least $1-\alpha$, contains no error. The model then regenerates from the first uncertified step. We test this on standard mistake-annotated benchmarks and on a new testbed of mechanistic chains from fibrosis biology. That testbed covers endothelial-to-mesenchymal transition (EndMT), the process by which blood-vessel lining cells lose their barrier identity and turn into scar-producing cells. Early mechanistic links in these chains decide every downstream therapeutic conclusion.

Core Claim

Prior debate systems decide what the answer is. We claim debate is better used to decide where a chain first breaks, and that this decision becomes trustworthy only when it is calibrated at the level of steps rather than final answers. Concretely, the claim has three parts. First, disagreement between heterogeneous agents about a single step is a stronger signal of that step being wrong than single-model self-review or sample divergence. Second, the signal improves when the skeptical agent must re-derive the step from the original question instead of reading the possibly corrupted prefix. Third, conformal thresholding turns these noisy scores into a prefix guarantee that holds on held-out data.

The object we calibrate is the first-error index $e^*$, the position of the earliest invalid step, with $e^* = \infty$ for a correct chain. For a threshold $\lambda$, let $\hat{k}(\lambda)$ be the number of steps accepted before the first step whose error score exceeds $\lambda$. We choose $\lambda$ on a calibration set so that

$$ \Pr\big(e^* \le \hat{k}(\lambda)\big) \le \alpha , $$

which says the accepted prefix contains an error at most an $\alpha$ fraction of the time. Raising $\lambda$ accepts longer prefixes, so the miss event only becomes more likely as $\lambda$ grows. That monotonicity is what makes standard conformal risk control applicable. The guarantee assumes calibration and test chains are exchangeable, meaning drawn from the same distribution. We treat violations of that assumption as an experimental question, not a footnote.

Relation to Prior Work

The closest multi-agent work already mixes debate with confidence, but at a different level and with different confidence sources. ReConcile runs a round-table of different model families and aggregates final answers by confidence-weighted voting. Its confidence comes from self-reported scores passed through a fixed rescaling. Consilience conformally calibrates which conversational move a group should make next (challenge, seek evidence, vote). Its guarantee is local, one move at a time. MAC removes the least-agreeing agent at each layer of a medical ensemble, dropping a fixed number of agents per layer. These papers motivate our work, because each shows that agreement structure carries signal. None of them calibrates a statement about which step of a reasoning chain is wrong. That is the quantity the error-location finding says matters.

Step-level debate exists in ChainLM, where personas debate each step before the generator proceeds, and adversarial skeptic/trusting roles appear in the Markov chain debate framework. We reuse the proponent-versus-skeptic structure from both. The difference is that they report final-answer accuracy, whereas we score localization and calibration directly against annotated first errors.

Two other papers supply method and constrain design. Deductive Verification of Chain-of-Thought Reasoning verifies each step using only the premises it cites. That isolation motivates our prefix-hidden skeptic. The same paper warns that isolation becomes intractable when every step depends on all prior steps, which is exactly the case for dense biological cascades. Mirror-Consistency shows that minority disagreement among samples from a single model improves calibration. It is our main single-model competitor.

Two findings from the calibration literature shape how we obtain confidence. Large Language Models Must Be Taught to Know What They Don't Know and Hao Liu et al. show two things: verbalized confidence is badly miscalibrated in open-ended settings, and small probes trained on hidden activations track correctness better. We therefore read each agent's step judgment through a probe, and we do not rely on its stated certainty.

Research Program

The system takes a completed chain of numbered steps. For each step $k$, a proponent agent defends the step given the prefix. A skeptic from a different model family then attacks it. In the main condition, the skeptic sees the original question and a summary of what step $k$ claims, but not steps $1$ through $k-1$. It must re-derive whether the claim follows from the problem. The point is to keep the skeptic from inheriting an earlier error, which is the snowball mechanism that Ling et al. describe. After one or two exchanges, each agent issues a verdict.

We extract three kinds of features from each debate over a step. The first is a linear probe score on each agent's hidden state at its verdict, trained on a small set of graded steps in the style of Kapoor et al. The second is the divergence between the agents' verdict distributions. The third is whether either side conceded. A small logistic model maps these features to a step error score. Conformal calibration on held-out annotated chains then sets $\lambda$ for a target $\alpha$. The chain is truncated at $\hat{k}$ and regenerated from there. Regeneration exploits the finding that models correct well once given a location.

The biology testbed is built from mechanistic cascades described in the fibrosis papers we have. Examples include disturbed flow or lipotoxicity leading to endothelial dysfunction (Lipotoxicity, the role in the process of atherosclerosis), and ER stress driving EMT (Melatonin Inhibits Endoplasmic Reticulum Stress and Epithelial-Mesenchymal Transition). Further links come from Smad3 signaling (Role of Smad3 signaling), SNAI1 recruiting HDAC1 (SNAI1 recruits HDAC1), and HDAC3/YY1 silencing RASAL1 (Epigenetic Suppression of RASAL1). The last stages cover barrier-marker loss and matrix deposition, as in cardiac fibrosis (Cathelicidin-related antimicrobial peptide protects against cardiac fibrosis in diabetic mice heart by regulating endothelial-mesenchymal transition) and pulmonary fibrosis (Alveolar epithelial cell dysfunction and epithelial-mesenchymal transition in pulmonary fibrosis pathogenesis).

Models generate chains that answer questions such as "why would HDAC3 inhibition be expected to reduce renal fibrosis?" Domain annotators then mark the first invalid step. We also create controlled corruptions, such as reversing a regulatory direction or swapping a marker, at a chosen depth.

EndMT raises one wrinkle that general benchmarks lack. Intermediate cell states in EMT shows that cells occupy hybrid, partly transitioned states. Some steps are therefore neither clearly right nor clearly wrong. Annotators get a third label, "overstated," and we report results both with that label treated as an error and with it treated as valid. That split tests whether calibration survives graded ground truth.

Baselines

The naive reference point accepts the whole chain with no localization. A second reference asks the generating model to name its own first error. The best reported number for that second reference is GPT-4's 52.87% first-error accuracy on BIG-Bench Mistake (LLMs cannot find reasoning errors). We rerun it with our own models so the comparison is on identical traces.

The strongest single-model competitor is Mirror-Consistency adapted to localization. We sample continuations from each prefix and score step $k$ by how much sampled continuations diverge from the chain at $k$. Its reported strength is on calibration of final answers: ECE (expected calibration error) of 0.039 versus 0.081 for plain self-consistency on GSM8K with GPT-3.5 (Mirror-Consistency). No reported localization number exists for it.

The strongest verification competitor is Natural Program step verification, which reports 84% step-verification accuracy on GSM8K with GPT-3.5-turbo (Deductive Verification).

A fine-tuned small critic is included because Tyen et al. report that a small model trained for mistake finding beat a larger generalist. The file gives no reported numeric result for that critic, so we train our own on the same annotation budget we give our probes.

MAC-style fixed-rank masking and ReConcile-style confidence-weighted voting are run as the existing debate methods.

Every method gets the same chains, the same calibration split, and the same matched token budget. Every method's raw scores pass through the same conformal step, so any difference comes from the scores, not from the thresholding. Beating a baseline means one of two things at matched cost. Either the method yields a longer average certified prefix at the same $\alpha$, or it achieves higher first-error localization accuracy with empirical miss rate at or below $\alpha$.

Ablations isolate the key ingredients. We compare prefix-hidden against prefix-exposed skeptics, heterogeneous against same-family agents, probe features against verbalized verdicts, and conformal against fixed thresholds.

Why not just…?

Why not just fine-tune a small critic to find the first error? This is the most serious threat. Tyen et al.'s evidence that trained critics beat larger generalists suggests it may work well in-domain. We run it with the same annotation budget and pass it through the same conformal step. If it matches debate on BIG-Bench Mistake and on biology, then the honest conclusion is that calibrated localization is what matters and debate is unnecessary. We would report that conclusion. Our hypothesis is that debate wins under the math-to-biology shift, where a critic trained on arithmetic traces has no domain grounding but heterogeneous agents re-deriving mechanisms do. The cross-domain split decides this.

Why not just use disagreement among samples from one model? Sample divergence is cheap, and Mirror-Consistency shows it carries calibration signal. However, ReConcile's echo-chamber finding implies that samples from one model share blind spots and will agree on a confidently wrong step. The heterogeneous-versus-same-family ablation measures exactly this. If sample divergence ties debate on localization, the multi-agent component is not justified.

Why not just ask each agent how confident it is? Kapoor et al. report open-ended verbalized-confidence ECE of roughly 30%, cut to 10% with training. We include verbalized verdicts as an ablation to show whether probes are needed.

Why not just verify each step in isolation with cited premises? Natural Program works on formal arithmetic. Its authors, however, flag dense interdependent chains as a failure case, and biological cascades are exactly that. We run it directly and expect its advantage to shrink from GSM8K to the EndMT set.

Proposed Experiments

The first experiment is BIG-Bench Mistake, which has 2,186 traces with annotated first errors. It tests whether debate localization exceeds the 52.87% self-localization reference and whether empirical miss rates track $\alpha$ across 0.05 to 0.2. The second applies the same pipeline to GSM8K chains, then measures final accuracy after regenerating from $\hat{k}$. The third is the EndMT testbed, with calibration fit on math only, on biology only, and on a mix. This quantifies how badly exchangeability breaks and whether a small in-domain calibration set repairs it. As a stress test for longer relational chains, we apply localization to 2-hop BioHopR reasoning. There o3-mini's reported precision is 14.57% (BioHopR), and we ask whether certified-prefix regeneration raises it.

Two checks are decisive. The first is whether the prefix-hidden skeptic beats the prefix-exposed one, which is our direct test of the snowball explanation. The second is whether debate beats the trained critic under domain shift at matched cost. If both checks fail, the proposal reduces to "conformally calibrate whatever step scorer is cheapest." That is still a useful result, but a different one from what we claim.

References

  1. LLMs cannot find reasoning errors, but can correct them given the error location
  2. ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs
  3. Consilience: Conformally Calibrated Communication Control for Hidden-Profile Multi-Agent Reasoning
  4. Adaptive Cluster Collaborativeness Boosts LLMs Medical Decision Support Capacity
  5. ChainLM: Empowering Large Language Models with Improved Chain-of-Thought Prompting
  6. Towards Detecting LLMs Hallucination via Markov Chain-based Multi-agent Debate Framework
  7. Deductive Verification of Chain-of-Thought Reasoning
  8. Mirror-Consistency: Harnessing Inconsistency in Majority Voting
  9. Large Language Models Must Be Taught to Know What They Don’t Know
  10. Uncertainty Calibration for Tool-Using Language Agents
  11. Lipotoxicity, the role in the process of atherosclerosis
  12. Melatonin Inhibits Endoplasmic Reticulum Stress and Epithelial-Mesenchymal Transition during Bleomycin-Induced Pulmonary Fibrosis in Mice
  13. [Corrigendum] Role of Smad3 signaling in the epithelial‑mesenchymal transition of the lens epithelium following injury
  14. SNAI1 recruits HDAC1 to suppress SNAI2 transcription during epithelial to mesenchymal transition
  15. Epigenetic Suppression of RASAL1 by HDAC3 and Cofactor YY1 Promotes Fibroblast–Myofibroblast Transition and Renal Fibrosis
  16. Cathelicidin-related antimicrobial peptide protects against cardiac fibrosis in diabetic mice heart by regulating endothelial-mesenchymal transition
  17. Alveolar epithelial cell dysfunction and epithelial-mesenchymal transition in pulmonary fibrosis pathogenesis
  18. Intermediate cell states in epithelial-to-mesenchymal transition
  19. BioHopR: A Benchmark for Multi-Hop, Multi-Answer Reasoning in Biomedical Domain