Breaking the Chain: A Causal Analysis of LLM Faithfulness to Intermediate Structures

Oleg Somov, Mikhail Chaichuk, Gleb Ershov, Karim Vafin, Mikhail Seleznyov, Alexander Panchenko, Elena Tutubalina

TL;DR

Schema-guided reasoning (SGR) pipelines ask LLMs to generate structured intermediates (like checklists or rubrics) to make their decisions controllable and interpretable. However, this paper reveals that while LLMs appear self-consistent, they often fail to update their final predictions when these intermediate structures are externally edited. Fortunately, this fragility can be largely eliminated by offloading the final decision mapping to an external tool or by fine-tuning the model using Direct Preference Optimization (DPO).

What Problem Does This Paper Address?

When deploying Large Language Models (LLMs) in high-stakes domains like medicine or legal compliance, practitioners often use Schema-Guided Reasoning (SGR). Instead of letting the model output a free-form chain-of-thought, SGR forces the model to fill out an explicit intermediate structure—such as a diagnostic checklist or a grading rubric—before making a final decision.

The core promise of SGR is controllability: if a human expert inspects the checklist and corrects an error, the final decision should update accordingly.

Figure 1: Causal faithfulness in schema-guided reason- ing. In high-stakes applications, domain experts (e.g., doctors, teachers) may edit the intermediate structure produced by an LLM such as a checklist or rubric. If the model is faithful, the final decision should update to reflect the edited mediator. This paper evaluates whether such interventions reliably change the model’s output.

But does this causal link actually hold? This paper investigates intervention faithfulness: whether an LLM genuinely relies on its intermediate structured representations as causal mediators, or if it simply treats them as decorative context while relying on hidden shortcuts directly from the input.

Methods

To rigorously evaluate this, the authors design a causal mediation protocol leveraging Pearl's front-door criterion. They specifically select tasks where a deterministic function maps the intermediate structure to the final decision (e.g., summing up rubric points to get a grade). Because the mapping is deterministic, any edit to the intermediate structure yields a single, unambiguous counterfactual ground truth.

The evaluation tests two primary intervention scenarios:

  1. Correction (COR): If the model generates an incorrect intermediate structure, the authors replace it with the ground-truth structure and check if the model fixes its final answer.
  2. Counterfactual (CNF): If the model generates a correct intermediate structure, the authors intentionally perturb one of its elements to see if the model shifts its final answer accordingly.

They evaluate two levels of faithfulness:

  • In-distribution ($F_{ID}$): Is the model's generated answer consistent with its own generated structure?
  • Strong Faithfulness ($F_{Strong}$): Does the model remain consistent after an external intervention alters the structure?

The gap ($\Delta = F_{ID} - F_{Strong}$) serves as the primary metric for unfaithfulness.

Datasets

The authors evaluate 12 instruction-tuned LLMs across four benchmarks representing distinct reasoning formats:

  • RiceChem: A chemistry grading task where the intermediate structure is a True/False rubric with weights, and the target is a recomputed numeric score.
  • AVeriTeC: A fact-checking dataset where claims are paired with binary sub-questions that dictate the final verdict.
  • TabFact: Table-based fact verification where the model generates a Domain Specific Language (DSL) query executed over table cells.
  • CRUXEval: A code execution prediction task where the model tracks intermediate variable assignments step-by-step to predict a function's output.

Key Insights and Ideas

  • Mediators as Context, Not Causes: LLMs treat intermediate structures as influential context rather than stable causal bottlenecks. They frequently bypass the structure, arriving at answers via direct shortcuts from the input.
  • Directional Asymmetry: Models are systematically more sensitive to counterfactual edits than corrections. It is much easier to disrupt a model's correct final answer by altering its checklist than it is to fix a wrong answer by correcting its checklist.
Figure 3: Symmetry analysis. The X-axis shows faithfulness under Correction interventions (where an incorrect mediator is replaced with a correct one), and the Y-axis shows faithfulness under Counterfactual interventions (and vice versa). Models with fewer than 10 generations in either subset are excluded due to noisy estimates.
  • Active Bypassing in Capable Models: On code reasoning (CRUXEval), larger and more capable models actually showed worse in-context faithfulness. They possess strong enough code-reasoning capabilities to compute the final answer directly from the source code, actively ignoring the intermediate execution trace they just generated.

Key Results

The Faithfulness Gap

Across three of the four datasets, the average gap between self-consistency ($F_{ID}$) and strong faithfulness ($F_{Strong}$) was a massive 18 to 23 percentage points, proving that apparent LLM faithfulness is highly fragile.

Tool Externalization Saves the Chain

The authors tested a "Tool Mode" where the deterministic mapping function was offloaded to an external execution environment (e.g., letting a Python script sum the rubric or execute the SQL query).

Figure 4: The bar plot shows the measured faithfulness gap for each model on the four datasets before and after tool use. The green arrows highlight this reduction, indicating the drop from the original gap to the post–tool-use gap for each model.

As shown in Figure 4, delegating this step to a tool nearly eliminated the faithfulness gap, bringing the residual gap below 0.03 for most configurations. This demonstrates that the unfaithfulness gap in standard pipelines is largely driven by the model's inability to reliably execute deterministic logic in-context, rather than a fundamental refusal to look at the mediator.

Prompting vs. Preference Optimization

The authors attempted to mitigate unfaithfulness by strengthening prompt instructions—even telling the model to treat the intermediate structure as the "ULTIMATE TRUTH." This yielded unstable and sometimes negative results.

Conversely, fine-tuning the models using Direct Preference Optimization (DPO) to penalize "stubborn" responses led to significant, systematic improvements. For example, Qwen-3 8B's strong faithfulness ($F_{Strong}$) on RiceChem jumped from 0.52 to 0.73 after DPO training.

Limitations and Future Work

A primary limitation is that this framework requires tasks with explicit, deterministic mappings from the intermediate structure to the final output. Many real-world reasoning tasks involve probabilistic or noisy mediators where a clean counterfactual target cannot be mathematically defined. Additionally, the study focused entirely on open-source models of moderate size (up to 32B/235B MoE); exploring how massive closed-source frontier models behave under similar causal interventions remains an open avenue for future research.

Summary updated

Abstract

In schema-guided reasoning (SGR) pipelines, LLMs produce explicit intermediate structures -- rubrics, checklists, or verification queries -- before committing to a final decision. SGR is increasingly adopted because it promises controllability: practitioners expect to inspect, edit, and override these structures to steer the outcome. But does the promise hold? We introduce a causal evaluation protocol to measure it: by selecting tasks where a deterministic function maps intermediate structures to decisions, every controlled edit implies a unique correct output. Across 12 models and 4 benchmarks, models appear self-consistent with their own intermediate structures but fail to update predictions after intervention -- revealing that apparent faithfulness is fragile once the intermediate structure changes. When derivation of the final decision from the structure is delegated to an external tool, this fragility largely disappears; stronger prompting yields only limited improvements, while preference optimization substantially improves intervention faithfulness. Overall, intermediate structures in schema-guided pipelines function as influential context rather than stable causal mediators.
BibTeX
@misc{somov2026breaking,
  author = {Oleg Somov and Mikhail Chaichuk and Gleb Ershov and Karim Vafin and Mikhail Seleznyov and Alexander Panchenko and Elena Tutubalina},
  title = {{Breaking the Chain: A Causal Analysis of LLM Faithfulness to Intermediate Structures}},
  year = {2026},
  eprint = {2603.16475},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.16475},
}
Figure