TL;DR
Researchers have developed a causal evaluation protocol to determine if Large Language Models (LLMs) actually use the intermediate reasoning structures they generate (like rubrics or proofs) to reach a final answer. The study reveals a troubling "faithfulness gap": while models rely on these structures more than surface-level text, they ignore logical edits to those structures over 50% of the time, treating their own "reasoning" as mere context rather than a causal roadmap.
What Problem Does This Paper Address?
As we move toward more complex AI applications, we often ask LLMs to show their work—generating rubrics for grading, checklists for fact-checking, or proof graphs for scientific reasoning. We assume these intermediate steps are faithful, meaning the model’s final decision is actually driven by that logic.
If a human intervenes and corrects a step in an AI’s proof, the final answer should change accordingly. If it doesn’t, the "reasoning" is just a decorative by-product—a post-hoc justification that doesn't actually mediate the decision. This paper investigates whether current LLMs are truly "steerable" through these structures or if they are simply taking hidden shortcuts to the answer.
Methods
The authors introduce a causal evaluation protocol based on the "front-door" adjustment framework. They test faithfulness using two primary types of interventions:
- Hold Structure, Vary Text (HSVT): They keep the reasoning structure (the "mediator") fixed but perturb or replace the input text. If the model is faithful to the structure, the final answer should remain the same despite the text changes.
- Local/Global Edit Consistency (LEC/GEC): They keep the input fixed but logically edit the intermediate structure (e.g., flipping a "True" to a "False" in a rubric). A faithful model should update its final decision to match the new logic.
The study evaluates nine modern models (including the Qwen, Llama, Falcon, and Gemma families) across two settings: Predicted (where the model generates its own structure) and Gold (where the model is given a ground-truth structure).
Datasets
The study utilizes four benchmarks that provide annotated intermediate structures:
- RiceChem: Chemistry grading using rubrics of True/False sub-steps.
- Averitec: Fact-checking where claims are verified via binary support questions.
- EntailmentBank: Multi-hop reasoning using structured proof graphs.
- TabFact: Table-based fact verification grounded in structured DSL queries.
Key Insights and Ideas
- Reasoning as Context, Not Logic: The most significant "aha moment" is that LLMs treat intermediate structures as context rather than mediators. They "read" the reasoning like background information rather than "executing" it like a program.
- The Generation Paradox: Surprisingly, models are more faithful to structures they generate themselves than to "Gold" structures provided by humans. The act of generation seems to bake the logic into the model's internal state more effectively than passive consumption.
- Global vs. Local Sensitivity: Models are much more likely to react to "global breaks" (destroying the entire proof) than "local edits" (changing one specific step). They tend to "smooth over" minor logical inconsistencies, treating them as noise.
- The Shortcut Bias: Larger models (like Qwen-235B) sometimes show lower faithfulness on certain metrics, suggesting that as models get more powerful, they develop more direct internal pathways (shortcuts) that bypass the explicit reasoning steps entirely.
Key Results
- High Reliance, Low Mediation: Models showed over 60% consistency (HSVT) when input text was changed, indicating they do look at the structure. However, they failed to update their answers under logical edits (LEC/GEC) more than 50% of the time.
- The Faithfulness Gap: In the EntailmentBank dataset, global faithfulness (GEC) dropped to under 15% for some configurations, meaning the model almost entirely ignored the fact that its proof had been broken.
- Self-Generated Superiority: Causal faithfulness was consistently 10–15% higher when models used their own generated structures compared to when they were provided with perfect ground-truth structures.
Limitations and Future Work
The authors acknowledge that their study focuses on "contextual faithfulness"—how the model uses information present in the prompt—rather than "parametric faithfulness" (what the model "knows" in its weights).
A major limitation is that current prompting and training methods do not explicitly enforce consistency between the reasoning chain and the final answer. The authors suggest that for AI to be truly collaborative and steerable, we need new training objectives that penalize models for ignoring their own intermediate logic. Until then, the "reasoning" we see in LLM outputs should be viewed with skepticism; it may look like a chain of logic, but for the model, it's often just more text.
Summary updated