To Trust or Not to Trust? Enhancing Large Language Models' Situated Faithfulness to External Contexts

Yukun Huang, Sanxing Chen, Hongyi Cai, Bhuwan Dhingra

TL;DR

This paper introduces the concept of situated faithfulness—the ability of an LLM to dynamically balance its internal knowledge against external context to resolve factual conflicts. The authors propose two frameworks, Self-Guided Confidence Reasoning (SCR) and Rule-Based Confidence Reasoning (RCR), alongside a novel alignment method, Confidence Reasoning Direct Preference Optimization (CR-DPO), which significantly boosts an LLM's ability to reject misleading information without sacrificing its ability to learn from correct context.

What Problem Does This Paper Address?

Retrieval-Augmented Generation (RAG) is widely used to reduce hallucinations by providing LLMs with external documents. However, what happens when the retrieved context is outdated, inaccurate, or intentionally malicious?

Currently, LLMs exhibit a form of "blind faith," often prioritizing external context over their own correct internal knowledge. This creates a knowledge conflict. To build truly robust systems, LLMs must exhibit situated faithfulness: they should trust their internal knowledge when it is correct and the external context is sketchy, but remain flexible enough to lean on external data when their internal knowledge is lacking or the external evidence is highly convincing.

Methods

The authors evaluate two primary paradigms for resolving knowledge conflicts:

  1. Self-Guided Confidence Reasoning (SCR): The model reasons directly in the text space.
  2. ImplicitSCR: The model is simply warned that the context might be incorrect and is prompted to use its judgment.
  3. ExplicitSCR: The model generates separate answers based on internal knowledge and external context, then uses a Chain-of-Thought (CoT) process to verbalize and weigh its confidence in both before delivering a final judgment.
  4. Rule-Based Confidence Reasoning (RCR): The model extracts explicit confidence signals (like token probabilities or self-evaluation scores) for both internal and external answers, and a predefined programmatic rule decides which answer to output.
Figure 1: Concept illustration of Self-guided Confidence Reasoning and Rule-based Confidence Reasoning. The goal is to enable the model to generate correct answers, regardless of input context accuracy, in contrast to direct input augmentation, which often blindly follows the context.

To supercharge smaller models that struggle with complex self-reflection, the authors introduce Confidence Reasoning Direct Preference Optimization (CR-DPO). This method trains models to prefer accurate, fact-checked confidence reasoning paths over flawed ones by optimizing self-sampled reasoning traces using preference learning.

Datasets

The authors benchmarked models across several diverse open-domain QA datasets (NaturalQA, TriviaQA, PopQA, FreshQA, and ClashEval), pairing questions with both correct retrieved online contexts and synthetically generated incorrect contexts.

Crucially, they introduced RedditQA, a new dataset consisting of human-written, in-the-wild incorrect contexts sourced from real Reddit posts containing factual errors. This dataset provides a realistic benchmark for how models handle genuine human misinformation rather than clean, machine-generated perturbations.

Key Insights and Ideas

  • Model Size Dictates Strategy: Stronger models like GPT-4o excel at text-space reasoning (SCR), naturally weighing evidence through chain-of-thought. Smaller models like Llama-3-8B lack these native reasoning capabilities and initially perform better under rigid, rule-based frameworks (RCR).
  • The Pitfalls of Rules: RCR approaches are heavily bottlenecked by noise and biases in confidence extraction. For instance, optimizing a model's token probability calibration does not automatically translate to better situated faithfulness because a well-calibrated score might still fail to align with a rigid decision rule.
  • Distillation Fails for Confidence Alignment: In a surprising twist, training Llama-3-8B on reasoning paths generated by GPT-4o actually hurt performance. Because confidence reasoning relies on a model understanding its own internal knowledge boundaries, the smaller model must learn from its own self-sampled reasoning paths rather than mimicking a stronger model with a completely different internal knowledge base.
Figure 2: An example where SCR fails to maintain situated faithfulness, but CR-DPO succeeds. In this case, the model’s internal answer is correct, but the context is misleading. Before training with CR-DPO, the model is misled despite correctly listing factual details. After training, it successfully arrives at the correct answer.

Key Results

  • GPT-4o Series: SCR methods achieved massive improvements over direct input augmentation baselines, boosting overall situated faithfulness by up to 24.2% on GPT-4o and 17.8% on GPT-4o mini.
  • Llama-3-8B: While standard ExplicitSCR underperformed on the smaller Llama model, applying CR-DPO yielded an average performance boost of 8.9% across all datasets.
  • Robustness to Strategy: CR-DPO demonstrated remarkable out-of-distribution generalization, successfully defending against highly deceptive context styles like "Reiteration" (where the wrong answer is repeated 10 times) and "Wiki-style" fabrications containing fake citations.

Limitations and Future Work

While the paper marks a significant step forward, the authors note several limitations:

  • Short-form Focus: The evaluation was limited to short-form QA where answers are concise phrases. Long-form generation introduces complex scenarios where internal and external knowledge might partially overlap or contradict across different sentences.
  • Single-Passage Constraint: The setup evaluated conflicts using only a single external passage at a time. Real-world RAG systems often retrieve multiple documents that may contradict each other as well as the model.
  • Inference Overhead: Explicit SCR and CR-DPO rely heavily on generating long Chain-of-Thought reasoning steps, which increases computational latency and cost during inference. Future work will focus on distilling these reasoning capabilities into more efficient, direct-generation frameworks.

Summary updated

Abstract

Large Language Models (LLMs) are often augmented with external contexts, such as those used in retrieval-augmented generation (RAG). However, these contexts can be inaccurate or intentionally misleading, leading to conflicts with the model's internal knowledge. We argue that robust LLMs should demonstrate situated faithfulness, dynamically calibrating their trust in external information based on their confidence in the internal knowledge and the external context to resolve knowledge conflicts. To benchmark this capability, we evaluate LLMs across several QA datasets, including a newly created dataset featuring in-the-wild incorrect contexts sourced from Reddit posts. We show that when provided with both correct and incorrect contexts, both open-source and proprietary models tend to overly rely on external information, regardless of its factual accuracy. To enhance situated faithfulness, we propose two approaches: Self-Guided Confidence Reasoning (SCR) and Rule-Based Confidence Reasoning (RCR). SCR enables models to self-assess the confidence of external information relative to their own internal knowledge to produce the most accurate answer. RCR, in contrast, extracts explicit confidence signals from the LLM and determines the final answer using predefined rules. Our results show that for LLMs with strong reasoning capabilities, such as GPT-4o and GPT-4o mini, SCR outperforms RCR, achieving improvements of up to 24.2% over a direct input augmentation baseline. Conversely, for a smaller model like Llama-3-8B, RCR outperforms SCR. Fine-tuning SCR with our proposed Confidence Reasoning Direct Preference Optimization (CR-DPO) method improves performance on both seen and unseen datasets, yielding an average improvement of 8.9% on Llama-3-8B. In addition to quantitative results, we offer insights into the relative strengths of SCR and RCR.
BibTeX
@inproceedings{huang2025trust,
  author = {Yukun Huang and Sanxing Chen and Hongyi Cai and Bhuwan Dhingra},
  title = {{To Trust or Not to Trust? Enhancing Large Language Models' Situated Faithfulness to External Contexts}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=K2jOacHUlO},
}
Figure