TL;DR
Schema-guided reasoning (SGR) pipelines ask LLMs to generate structured intermediates (like checklists or rubrics) to make their decisions controllable and interpretable. However, this paper reveals that while LLMs appear self-consistent, they often fail to update their final predictions when these intermediate structures are externally edited. Fortunately, this fragility can be largely eliminated by offloading the final decision mapping to an external tool or by fine-tuning the model using Direct Preference Optimization (DPO).
What Problem Does This Paper Address?
When deploying Large Language Models (LLMs) in high-stakes domains like medicine or legal compliance, practitioners often use Schema-Guided Reasoning (SGR). Instead of letting the model output a free-form chain-of-thought, SGR forces the model to fill out an explicit intermediate structure—such as a diagnostic checklist or a grading rubric—before making a final decision.
The core promise of SGR is controllability: if a human expert inspects the checklist and corrects an error, the final decision should update accordingly.
But does this causal link actually hold? This paper investigates intervention faithfulness: whether an LLM genuinely relies on its intermediate structured representations as causal mediators, or if it simply treats them as decorative context while relying on hidden shortcuts directly from the input.
Methods
To rigorously evaluate this, the authors design a causal mediation protocol leveraging Pearl's front-door criterion. They specifically select tasks where a deterministic function maps the intermediate structure to the final decision (e.g., summing up rubric points to get a grade). Because the mapping is deterministic, any edit to the intermediate structure yields a single, unambiguous counterfactual ground truth.
The evaluation tests two primary intervention scenarios:
- Correction (COR): If the model generates an incorrect intermediate structure, the authors replace it with the ground-truth structure and check if the model fixes its final answer.
- Counterfactual (CNF): If the model generates a correct intermediate structure, the authors intentionally perturb one of its elements to see if the model shifts its final answer accordingly.
They evaluate two levels of faithfulness:
- In-distribution ($F_{ID}$): Is the model's generated answer consistent with its own generated structure?
- Strong Faithfulness ($F_{Strong}$): Does the model remain consistent after an external intervention alters the structure?
The gap ($\Delta = F_{ID} - F_{Strong}$) serves as the primary metric for unfaithfulness.
Datasets
The authors evaluate 12 instruction-tuned LLMs across four benchmarks representing distinct reasoning formats:
- RiceChem: A chemistry grading task where the intermediate structure is a True/False rubric with weights, and the target is a recomputed numeric score.
- AVeriTeC: A fact-checking dataset where claims are paired with binary sub-questions that dictate the final verdict.
- TabFact: Table-based fact verification where the model generates a Domain Specific Language (DSL) query executed over table cells.
- CRUXEval: A code execution prediction task where the model tracks intermediate variable assignments step-by-step to predict a function's output.
Key Insights and Ideas
- Mediators as Context, Not Causes: LLMs treat intermediate structures as influential context rather than stable causal bottlenecks. They frequently bypass the structure, arriving at answers via direct shortcuts from the input.
- Directional Asymmetry: Models are systematically more sensitive to counterfactual edits than corrections. It is much easier to disrupt a model's correct final answer by altering its checklist than it is to fix a wrong answer by correcting its checklist.
- Active Bypassing in Capable Models: On code reasoning (CRUXEval), larger and more capable models actually showed worse in-context faithfulness. They possess strong enough code-reasoning capabilities to compute the final answer directly from the source code, actively ignoring the intermediate execution trace they just generated.
Key Results
The Faithfulness Gap
Across three of the four datasets, the average gap between self-consistency ($F_{ID}$) and strong faithfulness ($F_{Strong}$) was a massive 18 to 23 percentage points, proving that apparent LLM faithfulness is highly fragile.
Tool Externalization Saves the Chain
The authors tested a "Tool Mode" where the deterministic mapping function was offloaded to an external execution environment (e.g., letting a Python script sum the rubric or execute the SQL query).
As shown in Figure 4, delegating this step to a tool nearly eliminated the faithfulness gap, bringing the residual gap below 0.03 for most configurations. This demonstrates that the unfaithfulness gap in standard pipelines is largely driven by the model's inability to reliably execute deterministic logic in-context, rather than a fundamental refusal to look at the mediator.
Prompting vs. Preference Optimization
The authors attempted to mitigate unfaithfulness by strengthening prompt instructions—even telling the model to treat the intermediate structure as the "ULTIMATE TRUTH." This yielded unstable and sometimes negative results.
Conversely, fine-tuning the models using Direct Preference Optimization (DPO) to penalize "stubborn" responses led to significant, systematic improvements. For example, Qwen-3 8B's strong faithfulness ($F_{Strong}$) on RiceChem jumped from 0.52 to 0.73 after DPO training.
Limitations and Future Work
A primary limitation is that this framework requires tasks with explicit, deterministic mappings from the intermediate structure to the final output. Many real-world reasoning tasks involve probabilistic or noisy mediators where a clean counterfactual target cannot be mathematically defined. Additionally, the study focused entirely on open-source models of moderate size (up to 32B/235B MoE); exploring how massive closed-source frontier models behave under similar causal interventions remains an open avenue for future research.
Summary updated