Abstract
As autonomous agents and assistants, large language models (LLMs) often struggle with "hallucinations." Fundamentally, the problem is one of prioritization and balance: the LLM needs to understand or infer when it needs to be creative and balance that with its need to be accurate. Most efforts focus on either updating intrinsic knowledge via targeted post-training or by adding external knowledge sources which the LLM can reference neurosymbolically (e.g., via retrieval-augmented generation). However, these all eventually rely on the LLM's implicit reasoning ability during generation, still allowing for these random hallucinations despite high-quality training examples and references. Using aspect-oriented summarization as a case study, we propose LOgit REwriting (LORE), a new controlled generation paradigm which can simultaneously be faithful to external knowledge and to the LLM's intentions. LORE works by adding a rewriting module at left-to-right inference time, continuously reflecting on the newest prediction and trying to find a replacement that is more faithful to the source document. Then, it merges the logits of the replacement with those of the original prediction to generate the next token. We proposed a new long-context aspect-oriented summarization dataset, SLPAspect, and find that LORE generates 5.8% better summaries compared to the LLM without LORE-rewriting.
Introduction
Large Language Models (LLMs) have become indispensable tools for autonomous assistance, yet they remain plagued by "hallucinations"—the generation of plausible but factually incorrect information. This issue is particularly acute in high-stakes domains like medicine, education, and Speech-Language Pathology (SLP), where practitioners rely on accurate summaries of clinical literature to make evidence-based treatment decisions.
Even with advanced techniques like Retrieval-Augmented Generation (RAG), the fundamental tension between an LLM's parametric memory (creativity) and the requirement for strict adherence to source documents often leads to unfaithful outputs. If an LLM generates a summary where 95% of the text is highly coherent but 5% contains subtle hallucinations, a time-constrained clinician might take the entire output at face value, leading to real-world harm.
Methodology: The LORE Paradigm
To address this, we introduce LOgit REwriting (LORE), a novel decoding paradigm that ensures faithfulness by intervening during the left-to-right generation process. Unlike post-editing methods that attempt to fix errors after a full response is generated, LORE corrects the model token-by-token at the logit level.
The LORE pipeline consists of four primary stages:
- Segmentation & Attention Mapping: The source document is partitioned into segments. LORE utilizes the LLM's internal attention matrices to identify which segment the model is currently prioritizing during the generation of a specific token, effectively bypassing context-length limitations of smaller NLI models.
- NLI Grounding: The identified source segment (premise) and the current generation (hypothesis) are fed into an external Natural Language Inference (NLI) model (specifically,
roberta-large-mnli). - Gradient-Based Token Rewriting: If the NLI model identifies a lack of entailment, LORE backpropagates the grounding score $G$ to the NLI model's embedding layer. It then identifies alternative tokens in the vocabulary that would maximize the entailment score by calculating the cosine similarity between the gradient and token embeddings.
- Logit Mixing: The original LLM logits $M(I)$ are linearly interpolated with the rewritten logits $P$ using a tunable faithfulness parameter $\alpha$. The next token is then sampled from this improved distribution $M'(I)$.
Experimental Setup and Datasets
A significant contribution of this work is the introduction of SLPAspect, a large-scale, long-context aspect-oriented summarization dataset. It comprises over 4,000 scholarly articles from premier Speech-Language Pathology journals, featuring nearly 15,000 ground-truth summaries authored by the original researchers. This dataset provides a rigorous benchmark for evaluating faithfulness in specialized, high-risk domains. Additionally, we evaluate LORE on the AspectNews and ACLSum datasets.
Evaluation and Results
LORE demonstrates substantial improvements across both quantitative and qualitative metrics. On the SLPAspect dataset, applying LORE to a 1.5B parameter model (DeepSeek-R1-Distill-Qwen-1.5B) resulted in a dramatic increase in ROUGE-1 scores, rising from 22.7 to 32.5. Furthermore, LORE-augmented models consistently outperformed larger models and closed-source alternatives. In GPTScore evaluations (win rate vs. ground truth), the 1.5B LORE model achieved 71.0%, surpassing both its base version (67.1%) and GPT-4o-mini (68.0%).
Qualitative human and model-based assessments further validate LORE's efficacy. As shown in the comparison above, LORE's outputs were judged to be significantly more faithful (41% vs. 26%) and relevant (48% vs. 21%) compared to GPT-4 rewrites. This suggests that intervening at the logit level preserves the original intent of the generation while strictly enforcing grounding.
Discussion and Limitations
While LORE offers a powerful mechanism for reducing hallucinations, it has certain constraints. It is a "white-box" technique, requiring access to the model's internal logits and attention weights, which precludes its use with proprietary APIs. Additionally, the current implementation lacks a backtracking mechanism; it can only influence the current token, sometimes leading to redundant phrasing as the model attempts to reconcile its previous unfaithful tokens with the new, corrected path. Finally, the per-token optimization adds computational overhead, though this overhead decreases as a percentage of total compute as the base model size increases (e.g., ~14% for a 7B model).
Conclusion
LORE represents a significant step toward reliable, faithful LLM generation. By steering the model's internal probability distributions toward grounded tokens in real-time, LORE provides a robust defense against hallucinations in critical domains. Future work will explore adding deletion/backtracking capabilities and applying LORE to broader Retrieval-Augmented Generation (RAG) contexts.
Summary updated