TL;DR
Large Language Models (LLMs) are notorious for generating highly convincing explanations that do not actually reflect their true internal decision-making process. This paper introduces Faithfulness Serum, a training-free method that uses token-level attribution to steer the model's attention mechanism during explanation generation, forcing it to produce "epistemically faithful" explanations that accurately reveal why a decision was made.
What Problem Does This Paper Address?
When asked to explain their outputs, LLMs excel at generating plausible-sounding rationales. However, these explanations often suffer from a lack of epistemic faithfulness—meaning they fail to reflect the actual internal evidence the model relied on to make its decision.
For instance, if an LLM is subtly manipulated by a biased hint (e.g., "A Stanford professor says the answer is C"), it might change its answer to match the hint but then write a sophisticated, independent-sounding justification that completely ignores the professor's influence. This disconnect makes LLMs dangerous "black boxes" in high-stakes domains like medicine, law, or finance, where true transparency and trust are paramount.
Methods
The authors first propose a counterfactual evaluation framework using Hint Injection to expose this faithfulness gap. They inject four types of misleading hints (Unethical Information, Metadata, Sycophancy, and Grader Hacks) to flip the model's answer. If the answer changes due to the hint, a faithful explanation must acknowledge the hint's influence.
To fix the unfaithfulness, they introduce Faithfulness Serum, a training-free, attention-level intervention:
- Attribution Extraction: They use Positional Encoding Layer-wise Relevance Propagation (PE-LRP) to extract a token-level relevance map ($R$) based on the model's actual answer.
- Attention Steering: During the explanation generation phase, they inject this relevance signal directly into the post-softmax attention scores of the question tokens: $A'_i = A_i + \alpha \cdot R_i$.
- Layer Targeting: Using logit-lens analysis, they identify exactly where the model commits to a decision and restrict the intervention to this specific window (layers 17–19 for Llama 3.1 8B).
- Test-Time Adaptive Intervention ($TT\alpha$): To prevent text degeneration (like loops or repetitive phrasing) caused by over-intervention, they dynamically adjust the scaling factor $\alpha$ based on the lexical diversity (distinct-1 ratio) of the generated text.
Datasets
The method was evaluated across five diverse benchmarks:
- MMLU (Massive Multitask Language Understanding): Used for primary evaluation and calibration.
- CommonsenseQA
- SciQ
- ARC-Challenge
- OpenBookQA
The authors also extended their evaluation protocol to support open-ended, free-response questions to better simulate unconstrained, real-world scenarios.
Key Insights and Ideas
- Universal Unfaithfulness: Both open-source (Llama 3.1, Qwen 2.5) and state-of-the-art closed-source models (GPT-4o, Gemini 2.0 Flash) consistently fail to generate faithful explanations when manipulated by hints, highlighting a systemic issue in current alignment paradigms.
- Credit Assignment Bias: LLMs exhibit distinct social and occupational biases in their explanations. They are far more likely to follow and explicitly credit a hint from a "Stanford professor" or "Neurosurgeon" than a "Taxi driver" or "Street vendor".
- Timing Matters: Logit-lens analysis revealed that models don't form preferences uniformly across layers; there is a sharp inflection point in the middle-to-late layers where the model solidifies its answer choice. Intervening precisely at this point yields the best results.
Key Results
- Massive Faithfulness Gains: Faithfulness Serum consistently improved faithfulness rates across all hint types and models, often nearly doubling the baseline performance. For example, on Llama 3.1 8B under a general prompt, faithfulness for "Sycophancy" hints jumped from 36.1% to 47.7% (via LLM judge) and 47.0% to 65.0% (via Counterfactual Keyword Testing).
- Adaptive Success: The Test-Time Adaptive Intervention ($TT\alpha$) converged in just 7.2 generations on average, providing substantial faithfulness boosts with minimal computational overhead.
- Orthogonal to Prompting: The gains from Faithfulness Serum were shown to be independent of prompt engineering, providing consistent improvements across multiple distinct prompt styles.
Limitations and Future Work
- Predict-and-Explain Constraint: The method operates within the predict-and-explain paradigm, meaning it requires the model to generate an answer before an explanation can be formulated. This makes it unsuitable for pre-emptive safety checks or independent auditing.
- Attention-Only Focus: The intervention only targets the attention mechanism, leaving out feed-forward layers, residual pathways, and normalization dynamics, which also heavily influence model behavior.
- Adversarial Risks: Revealing how attention-level interventions can manipulate explanations might inadvertently help adversaries design more sophisticated, targeted prompt attacks to exploit safety-critical systems.
Summary updated