Intervening on a Model's Own Reasoning to Find the Hallucinations That Matter

When a language model writes out a multi-step answer, some of its intermediate claims are false and some are true. What matters more is which of those claims the final answer actually depends on. A fabricated premise that the answer ignores is harmless to the answer but dishonest. A slightly wrong number that every later step inherits is fatal. Current hallucination detectors cannot tell these cases apart. They either score every claim with equal weight, or they propagate a heuristic "trust" score along a dependency graph that the model itself wrote and that nobody has checked.

This note proposes treating the model's reasoning trace as a structural causal model, meaning a set of variables where each one is produced by a function of its declared parents plus some unexplained noise. The declared dependencies then become claims that can be tested by intervention. Concretely, we edit one intermediate step to a counterfactual value, hold the decoding randomness and all non-dependent text fixed, regenerate only the steps that claim to depend on the edited one, and watch what happens to the answer. This yields two measurements that no existing detector produces. The first is whether a step is load-bearing: changing it changes the answer. The second is whether a declared dependency is real: changing the parent changes the child. With these we can spend expensive fact-checking only on load-bearing steps, flag answers whose stated premises are not their actual causes, and repair a bad step by regenerating only what descends from it.

Core Claim

The closest system is the Structural Reasoning Model of Hallucination Detection from Structural Reasoning Model. It prompts an LLM to emit conditions, steps, and explicit parent sources, checks each step with a judge that sees only the declared parents, and aggregates the results with FlowScore, a propagation of "logic-mass" through the graph. It reaches 76.59% average precision on MATH and 83.08% on HumanEval. It also takes the declared graph at face value and pays one judge call per step. Our claim is that the declared graph is the weakest link. Models' own explanations are often unfaithful rationalizations, as Causal Articulation Theory (CAT) argues when it derives explanations from true SCM parameters instead. Interventions on the trace can check the graph and rank steps by causal importance in the same pass. The resulting detector should be at least as accurate as FlowScore at lower verification cost. It should also catch a failure FlowScore cannot see: the "accuracy-honesty decoupling" documented by Tracing the Cascade (SCHEMA), in which models reach correct answers through hallucinated intermediate steps.

Two definitions carry the method. Let $X_1,\dots,X_n$ be the trace's steps and $A$ the final answer, produced under fixed decoding noise $U$. Here "fixed noise" means greedy decoding or a fixed sampling seed. Under that noise the observed answer is $a$. Let $\tilde{x}_i^{(k)}$, $k=1..K$, be minimal counterfactual edits of step $i$: a changed number, a swapped entity, or a mutated expression. The load-bearing score of step $i$ is

$$\delta_i = \frac{1}{K}\sum_{k=1}^{K} \mathbb{1}\left[A_{do(X_i=\tilde{x}_i^{(k)}),\,U} \neq a\right],$$

where the subscript means the answer obtained after setting step $i$ to the edit, keeping $U$ and all non-descendants of $i$ verbatim, and regenerating only the declared descendants. For a declared edge $j \to i$, the edge-faithfulness score $\epsilon_{j\to i}$ is defined the same way. It is the fraction of edits to $X_j$ under which the regenerated $X_i$ changes its key value.

Fixing $U$ and the non-descendant text is the counterfactual step of Pearl's ladder, the "infer the noise, then change, then re-propagate" procedure formalized by Halpern and Pearl. It is also what separates our intervention from simply resampling everything after step $i$.

How This Differs From Prior Causal Work on Hallucination

Several papers bring causal machinery to hallucination, but they place it elsewhere. Mitigating Hallucinations via Causal Reasoning fine-tunes models to build and reason over causal graphs of the problem, through CDCR-SFT on 25,368 synthesized samples, with a 10% gain on HaluEval. The SCM there is content the model reasons about, not an object we audit.

Conditional Front-Door Adjustment treats the whole chain of thought as one mediator node and intervenes only on input context by swapping entities. We intervene inside the chain, one step at a time. CI4MRC masks name-sensitive neurons in encoder models. Aspect-Based Causal Abstention estimates query-level effects with inverse probability weighting and responds by abstaining. Neither localizes a faulty mechanism within a generation.

SCHEMA comes closest in spirit. It asks "had step $t$ been corrected, would the answer be right?" But it does this offline, by manually replacing step text, against an external concept graph. We make the same question automatic, run it against the model's own declared graph, and use it at inference time to allocate verification and drive repair.

There is also a constraint from Sun and Yang: forcing source attribution slightly hurt fact-heavy HotpotQA. This is a real risk to any structured-trace method, and our experiments measure it directly instead of assuming it away.

Research Program

The pipeline has four stages. The first is structured generation in SRM format. The second is counterfactual edit generation. Edits are produced by rule in typed domains: numeric perturbation for MATH, operator and constant mutation for HumanEval, and same-type entity swap for multi-hop QA. In free text, an auxiliary model proposes minimal edits that a checker confirms are value-changing.

The third stage runs interventions. Each step gets $\delta_i$, and edges get $\epsilon$ scores on a sampled subset. Only descendants are regenerated, so an intervention costs tokens roughly proportional to the size of its subgraph.

The fourth stage acts on the scores. An external judge or retrieval check is applied only to the top-$\delta$ steps under a fixed call budget. An answer is flagged as a shortcut when its supported premises have low $\delta$ while its answer stays fixed under edits to every stated premise. This signature is the decoupling: the answer is coming from somewhere the trace does not admit, plausibly a memorized prior. When a load-bearing step fails verification, we repair it with $do(X_i=\text{grounded value})$ and regenerate its descendants.

One analysis question matters especially. Measured edge faithfulness may turn out low. Interventions might change steps declared independent, or leave declared children unchanged. If so, FlowScore is propagating trust along edges the model does not use, and that would explain where it fails.

Baselines

The naive reference point is the base model's unverified answer with greedy decoding, together with self-consistency over the same number of samples our method spends on interventions. Sun and Yang report that sampling-based detection reaches only about 25% average precision on HumanEval, so self-consistency should be weak on code. It remains the honest "spend the same tokens on more samples" control.

The strongest existing detector is SRM with FlowScore, which reports 76.59% AP on MATH and 83.08% AP on HumanEval in Sun & Yang. We reimplement it on the same base model, the same structured traces, and the same judge model, and we report both AP and judge calls per problem. Beating it means matching or exceeding its AP while using a fixed fraction, targeted at no more than half, of its judge calls. A clear AP gain at equal calls would count as an even stronger result.

For span-level hallucination on RAGTruth, the strongest reported detector is RL4HS-7B at 55.9 Span-F1, against 50.1 for SFT and 51.2 for o3, in Learning to Reason for Hallucination Span Detection. We do not expect to beat it at localizing spans; that is not our claim. Instead, we test whether weighting its detected spans by $\delta$ better predicts which spans actually corrupt the response's main conclusion.

For repair on multi-hop QA, the strongest causal baseline is CFD-Prompting. The file reports only relative gains for it, +2.71 EM and +4.54 F1 over earlier causal methods. The file gives no absolute HotpotQA or MuSiQue score for it, so we rerun it on the same base model.

Two ablations isolate the key ingredient. Truncate-and-resample discards everything after the edited step instead of only descendants, which tests whether fixing noise and non-descendants matters. Random-step verification under the same call budget tests whether $\delta$ ranking matters.

Why not just…?

Why not just have a judge check every step and regenerate from the first flagged one? This is the most serious threat. If per-step judging is accurate and affordable, the causal machinery adds little to detection. Our claim rests on two things this approach lacks. First, it spends calls on steps that do not affect the answer. Second, it cannot detect decoupled answers, where every stated step checks out but the answer comes from elsewhere. The budgeted comparison against FlowScore settles the first point, and the shortcut diagnostic below settles the second. For repair, truncate-and-resample is exactly our ablation. If descendant-only regeneration is not better, we drop that part of the claim.

Why not just weight steps by position or graph centrality instead of intervening? Hallucinations cluster at the tail of long generations, as shown in RAGTruth, and at high-degree hubs, as shown by SCHEMA. Cheap heuristics such as "verify the last $k$ steps" or "verify the highest out-degree nodes" might therefore approximate $\delta$. We include both as budget-matched rankers. If either matches $\delta$-ranking within noise, we will report that interventions are unnecessary for allocation. In that case the remaining contribution would be the edge-faithfulness and shortcut tests, which heuristics cannot provide.

Why not just train a span detector with RL? Trained detectors such as RL4HS and HDM-4B-RL from From Detection to Diagnosis find wrong text well. They are rewarded for span overlap, though, not for consequence. They also say nothing about whether a correct-looking answer was produced by its stated reasoning. Our method is complementary: it can sit on top of any span detector. The RAGTruth experiment tests whether combining the two beats the detector alone at predicting response-level failure.

Proposed Experiments

Detection runs on MATH and HumanEval with an open model in the Llama-3.1-8B or Qwen2.5-7B class, plus one API model. The metrics are answer-error AP versus judge-call budget, along with total tokens including intervention cost.

The decisive check for decoupling uses MuSiQue and HotpotQA. Following Zhao et al., we swap a bridge entity in the supporting context so that the gold answer changes. An answer that does not move while its trace cites the swapped fact is, by construction, a shortcut. We measure how well our shortcut flag recovers these cases, and compare against FlowScore and self-consistency, neither of which has any signal for them. Applying the same flag to SCHEMA's protein-domain trajectories tests whether low-$\delta$ correct answers coincide with SCHEMA's decoupled cases. Since the file reports no exact SCHEMA severity values, these comparisons will be rerun from scratch.

Repair is measured as EM and F1 on HotpotQA and MuSiQue against CFD-Prompting and truncate-and-resample. We separately report the structured-prompting penalty Sun and Yang observed on simple fact questions.

The project fails cleanly under two conditions. It fails if $\delta$-guided verification does no better than tail or centrality heuristics at equal budget. It also fails if the shortcut flag cannot separate the entity-swap shortcuts from genuine reasoning. Either outcome would show that the model's trace is not usefully a causal object at the granularity we chose.