Context-Fidelity Boosting: Enhancing Faithful Generation through Watermark-Inspired Decoding

Weixu Zhang, Fanghua Ye, Qiang Gao, Jian Li, Haolun Wu, Yuxing Tian, Sijing Duan, Nan Du, Xiaolong Li, Xue LiuarXiv 2026 · arXiv

TL;DR

Large Language Models (LLMs) often suffer from "faithfulness hallucinations," where they prioritize their internal training data over the specific context provided by a user. Context-Fidelity Boosting (CFB) is a lightweight, decoding-time framework that uses logit-shaping—a technique borrowed from text watermarking—to nudge the model toward generating tokens supported by the input context, significantly improving faithfulness without the need for retraining.

What Problem Does This Paper Address?

Even the most advanced LLMs struggle with a "tug-of-war" between their parametric memory (knowledge acquired during pre-training) and the provided context (e.g., a document in a RAG pipeline). When these two conflict, models often generate "faithful hallucinations"—outputs that are fluent and plausible but factually inconsistent with the input text.

For example, if a model was trained on data up to 2022, it might insist the last Olympics were in Tokyo, even if provided with a context document about the 2024 Paris Games. Existing solutions often require expensive fine-tuning or brittle prompt engineering. The authors sought a model-agnostic, efficient way to intervene during the decoding process to ensure the model "stays on script."

Figure 1: Illustration of context-faithful decoding: Tra- ditional decoding relies on parametric knowledge (fa- voring “Tokyo”), while our logit-shaping approach dy- namically adjusts token probabilities to better align with the given context about “Paris 2024”.

Methods

CFB introduces additive logit adjustments based on how well a token is supported by the source context. The authors propose three levels of increasing adaptivity:

  1. Static Boosting: Applies a fixed bias ($\delta$) to all tokens that appear in the source context. It’s simple and efficient but doesn't distinguish between important and irrelevant tokens.
  2. Context-Aware Boosting: Scales the bias dynamically using the Jensen-Shannon Divergence (JSD) between the model's next-token distribution with context vs. without context. If the context significantly changes the model's preference, the boost is increased.
  3. Token-Aware Boosting: The most granular approach. It redistributes the adaptive boost across tokens based on local relevance signals, specifically aggregating attention scores over source positions and calculating semantic similarity using token embeddings.
Figure 2: Overview of the proposed Context-Fidelity Boosting (CFB) framework. CFB applies additive logit shaping through three strategies of increasing adaptivity: (1) Static Boosting, which uniformly increases logits of source-supported tokens by a fixed value; (2) Context-Aware Boosting, which scales the boost using the divergence between context-aware and context-free next-token distributions; and (3) Token-Aware Boosting, which further redistributes the adaptive boost according to token-level relevance estimated from attention over source positions and source-scoped semantic similarity.

Importantly, CFB requires no architectural changes or retraining, making it a "plug-and-play" solution for open-weight models.

Datasets

The researchers evaluated CFB across two primary tasks using several state-of-the-art LLMs (Llama2-13B, Llama3-8B, and Mistral-7B):

  • Summarization: Evaluated on CNN-DM and XSum. Metrics included ROUGE-L (quality), FactKB (consistency), and BERT-P (semantic preservation).
  • Question Answering: Evaluated on NQ-Synth (where context aligns with parametric knowledge) and NQ-Swap (where context intentionally conflicts with parametric knowledge).

Key Insights and Ideas

  • Watermarking as Inspiration: The core "aha moment" is repurposing logit-shaping—usually used to embed hidden signals in text for detection—to instead steer the model toward faithfulness.
  • The Stability of Semantics: Ablation studies revealed that semantic similarity is the most critical component for the stability of the token-aware variant. Removing it caused a severe performance collapse, while removing attention only led to a modest drop.
  • Complementary vs. Conflicting Knowledge: CFB is exceptionally effective when the context complements the model's knowledge (NQ-Synth). However, in high-conflict scenarios (NQ-Swap), contrastive methods that explicitly suppress parametric priors (like ADACAD) still hold an edge.

Key Results

CFB consistently outperformed strong decoding-time baselines like Context-Aware Decoding (CAD) and ADACAD in summarization tasks.

  • Summarization: On CNN/DM, Token-aware CFB achieved the best ROUGE-L (34.52) and FactKB (96.87) for Mistral-7B. For Llama2-13B, Static CFB reached a high FactKB of 98.85.
  • Efficiency: Static and Context-aware variants are incredibly efficient, adding less than 0.003% to the base model's FLOPs. While Token-aware CFB is more intensive, it remains practical for real-world inference.
  • Human Evaluation: In qualitative tests, Token-aware CFB received the highest human ratings for faithfulness (4.31/5) and informativeness (4.12/5), beating out all baselines while maintaining comparable fluency.
Table 2: Main summarization results on CNN/DM and XSum. We report ROUGE-L, FactKB, and BERT-P. Best results for each model are shown in bold.

Limitations and Future Work

The authors acknowledge a few hurdles:

  • White-Box Requirement: CFB requires access to model internals (logits, attention, embeddings), making it currently incompatible with black-box APIs like GPT-4 or Claude.
  • Conflict Handling: While great for grounding, the method is less dominant in scenarios where the context and the model's internal knowledge are in direct, aggressive conflict.
  • Future Directions: The team suggests exploring black-box approximations and more robust token-level relevance modeling to further reduce decoding costs.

Summary updated

Abstract

Large language models (LLMs) often produce content that contradicts or overlooks information provided in the input context, a phenomenon known as faithfulness hallucination. In this paper, we propose Context-Fidelity Boosting (CFB), a lightweight and general decoding-time framework that reduces such hallucinations by increasing the generation probability of source-supported tokens. Motivated by logit-shaping principles from watermarking techniques, CFB applies additive token-level logit adjustments based on a token's degree of support from the input context. Specifically, we develop three boosting strategies: static boosting, which applies a fixed bias to source-supported tokens; context-aware boosting, which scales this bias using the divergence between next-token distributions with and without context; and token-aware boosting, which further redistributes the adaptive bias according to local relevance estimated from source-position attention and source-scoped semantic similarity. CFB requires no retraining or architectural changes, making it compatible with a wide range of LLMs. Experiments on summarization and question answering tasks across multiple open-source LLMs show that CFB consistently improves faithfulness metrics with minimal generation overhead. Our implementation is fully open-sourced.
BibTeX
@misc{zhang2026context,
  author = {Weixu Zhang and Fanghua Ye and Qiang Gao and Jian Li and Haolun Wu and Yuxing Tian and Sijing Duan and Nan Du and Xiaolong Li and Xue Liu},
  title = {{Context-Fidelity Boosting: Enhancing Faithful Generation through Watermark-Inspired Decoding}},
  year = {2026},
  eprint = {2604.22335},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.22335},
}

Loading related content...

Figure