TL;DR
This paper introduces the concept of situated faithfulness—the ability of an LLM to dynamically balance its internal knowledge against external context to resolve factual conflicts. The authors propose two frameworks, Self-Guided Confidence Reasoning (SCR) and Rule-Based Confidence Reasoning (RCR), alongside a novel alignment method, Confidence Reasoning Direct Preference Optimization (CR-DPO), which significantly boosts an LLM's ability to reject misleading information without sacrificing its ability to learn from correct context.
What Problem Does This Paper Address?
Retrieval-Augmented Generation (RAG) is widely used to reduce hallucinations by providing LLMs with external documents. However, what happens when the retrieved context is outdated, inaccurate, or intentionally malicious?
Currently, LLMs exhibit a form of "blind faith," often prioritizing external context over their own correct internal knowledge. This creates a knowledge conflict. To build truly robust systems, LLMs must exhibit situated faithfulness: they should trust their internal knowledge when it is correct and the external context is sketchy, but remain flexible enough to lean on external data when their internal knowledge is lacking or the external evidence is highly convincing.
Methods
The authors evaluate two primary paradigms for resolving knowledge conflicts:
- Self-Guided Confidence Reasoning (SCR): The model reasons directly in the text space.
- ImplicitSCR: The model is simply warned that the context might be incorrect and is prompted to use its judgment.
- ExplicitSCR: The model generates separate answers based on internal knowledge and external context, then uses a Chain-of-Thought (CoT) process to verbalize and weigh its confidence in both before delivering a final judgment.
- Rule-Based Confidence Reasoning (RCR): The model extracts explicit confidence signals (like token probabilities or self-evaluation scores) for both internal and external answers, and a predefined programmatic rule decides which answer to output.
To supercharge smaller models that struggle with complex self-reflection, the authors introduce Confidence Reasoning Direct Preference Optimization (CR-DPO). This method trains models to prefer accurate, fact-checked confidence reasoning paths over flawed ones by optimizing self-sampled reasoning traces using preference learning.
Datasets
The authors benchmarked models across several diverse open-domain QA datasets (NaturalQA, TriviaQA, PopQA, FreshQA, and ClashEval), pairing questions with both correct retrieved online contexts and synthetically generated incorrect contexts.
Crucially, they introduced RedditQA, a new dataset consisting of human-written, in-the-wild incorrect contexts sourced from real Reddit posts containing factual errors. This dataset provides a realistic benchmark for how models handle genuine human misinformation rather than clean, machine-generated perturbations.
Key Insights and Ideas
- Model Size Dictates Strategy: Stronger models like GPT-4o excel at text-space reasoning (SCR), naturally weighing evidence through chain-of-thought. Smaller models like Llama-3-8B lack these native reasoning capabilities and initially perform better under rigid, rule-based frameworks (RCR).
- The Pitfalls of Rules: RCR approaches are heavily bottlenecked by noise and biases in confidence extraction. For instance, optimizing a model's token probability calibration does not automatically translate to better situated faithfulness because a well-calibrated score might still fail to align with a rigid decision rule.
- Distillation Fails for Confidence Alignment: In a surprising twist, training Llama-3-8B on reasoning paths generated by GPT-4o actually hurt performance. Because confidence reasoning relies on a model understanding its own internal knowledge boundaries, the smaller model must learn from its own self-sampled reasoning paths rather than mimicking a stronger model with a completely different internal knowledge base.
Key Results
- GPT-4o Series: SCR methods achieved massive improvements over direct input augmentation baselines, boosting overall situated faithfulness by up to 24.2% on GPT-4o and 17.8% on GPT-4o mini.
- Llama-3-8B: While standard ExplicitSCR underperformed on the smaller Llama model, applying CR-DPO yielded an average performance boost of 8.9% across all datasets.
- Robustness to Strategy: CR-DPO demonstrated remarkable out-of-distribution generalization, successfully defending against highly deceptive context styles like "Reiteration" (where the wrong answer is repeated 10 times) and "Wiki-style" fabrications containing fake citations.
Limitations and Future Work
While the paper marks a significant step forward, the authors note several limitations:
- Short-form Focus: The evaluation was limited to short-form QA where answers are concise phrases. Long-form generation introduces complex scenarios where internal and external knowledge might partially overlap or contradict across different sentences.
- Single-Passage Constraint: The setup evaluated conflicts using only a single external passage at a time. Real-world RAG systems often retrieve multiple documents that may contradict each other as well as the model.
- Inference Overhead: Explicit SCR and CR-DPO rely heavily on generating long Chain-of-Thought reasoning steps, which increases computational latency and cost during inference. Future work will focus on distilling these reasoning capabilities into more efficient, direct-generation frameworks.
Summary updated