When training a model to summarize an article, answer a question, or describe a structured table, we operate under a fundamental assumption: the "gold standard" target text is a faithful, accurate representation of the source input. Supervised fine-tuning treats these human-written or heuristically gathered targets as absolute truth, penalizing models whenever their predictions deviate from the reference.
However, systematic analyses of popular benchmarks reveal a stark reality. Reference texts frequently contain extrinsic noise—information, opinions, clickbait, or stylistic flair that cannot be logically derived from the source document. When models are forced to mimic these flawed targets, they don't just learn how to summarize; they learn how to hallucinate.
The Illusion of the Gold Standard
To understand why models invent facts, we must first look at how our training data is constructed. Many large-scale datasets rely on distant supervision or web scraping. For instance, news headlines or introductory highlights are often used as proxy summaries for an entire article.
As Noam Dahan notes in The State and Fate of Summarization Datasets: A Survey, these proxy labels are optimized for reader engagement and click-through rates rather than factual synthesis. They frequently introduce external context or sensationalized framing absent from the body text.
Even human-curated datasets are heavily contaminated. In Analyzing and Evaluating Faithfulness in Dialogue Summarization, Bin Wang 0040 et al. discovered that up to 17% of human-written reference summaries in dialogue datasets are factually inconsistent with the source text. Humans naturally inject background knowledge, assumptions, or unmentioned details into their writing. This divergence is not unique to text-to-text tasks; Zhenyi Wang demonstrates in Towards Faithful Neural Table-to-Text Generation with Content-Matching Constraints that open-domain table-to-text references regularly include biographical facts that cannot be found anywhere in the provided structured data.
Why Standard Training Forces Hallucinations
Standard autoregressive language models are trained using Maximum Likelihood Estimation (MLE), which minimizes the cross-entropy loss between the model's output and the target text:
$$\mathcal{L} = -\sum_{t=1}^{T} \log P(y_t \mid y_{<t}, X)$$
Where $X$ is the source document and $y_t$ is the target token at step $t$.
Consider what happens when the target sequence contains an extrinsic fact—such as a person's age or a specific location—that never appeared in the source $X$. Because the model cannot logically deduce this token from $X$, its predictive probability $P(y_t \mid y_{<t}, X)$ will initially be near zero, resulting in a massive loss penalty.
To minimize this loss during backpropagation, the model's weights are forced to adjust so it can "guess" these ungrounded tokens. As Tanya Goyal et al. demonstrate in Training Dynamics for Text Summarization Models, models learn basic copying behaviors very early in training, but begin to emulate these extrinsic discrepancies later in the fine-tuning process. The model essentially learns a dangerous meta-strategy: to achieve a low loss, I must spontaneously invent plausible-sounding details.
Breaking the Mimicry Loop via Loss Truncation
If our datasets are fundamentally noisy, how can we train a model to be strictly faithful? The key intuition lies in turning the model's own uncertainty into a filter.
When a target token requires external knowledge that isn't in the source, the model cannot predict it accurately, causing a localized spike in the loss function. Conversely, tokens that are cleanly grounded in the source text yield much lower loss values.
Daniel Kang et al. leverage this phenomenon in Improved Natural Language Generation via Loss Truncation. Instead of forcing the model to optimize for every single token in the dataset, their framework adaptively drops the highest-loss examples during a gradient step.
Standard MLE:
[Source Text] ---> [Model] ---> Calculate Loss on ALL Target Tokens ---> Update Weights
Loss Truncation:
[Source Text] ---> [Model] ---> Calculate Token Losses ---> Drop High-Loss Tokens (Noise) ---> Update Weights
By dynamically truncating the loss distribution, the algorithm ignores the ungrounded, hallucinated phrases that the model struggles to predict. This prevents the model from updating its weights to mimic extrinsic noise, resulting in a system that remains strictly bounded by the provided context.
Filtering, Reclassifying, and Consensus
Beyond modifying the loss function, researchers have developed structural training paradigms to handle reference-source divergence.
Explicit Data Sanitation
One straightforward approach is to clean the dataset before training ever begins. In Questioning the Validity of Summarization Datasets and Improving Their Factual Consistency, Yanzhu Guo et al. use automated factuality metrics to evaluate the alignment between source documents and their reference summaries. By pruning the ungrounded pairs entirely to create a smaller, cleaner dataset (SummFC), they show that models trained on less data can actually achieve significantly higher factual consistency.
Contrastive Learning with Flawed References
Rather than discarding noisy data, we can change how the model perceives it. In FaMeSumm: Investigating and Improving Faithfulness of Medical Summarization, Nan Zhang et al. identify unfaithful human references and reclassify them as negative examples within a contrastive learning framework. Instead of treating the noisy summary as a target to emulate, the model is explicitly penalized for matching it, teaching the network exactly what kind of ungrounded information to avoid.
Democratic Consensus
Instead of anchoring training to a single, potentially flawed human reference, models can learn from a consensus of generations. Junhyun Lee et al. introduce this in BRIDO: Bringing Democratic Order to Abstractive Summarization. By optimizing the model to favor the shared, overlapping factual agreements among multiple generated candidates rather than a single noisy gold standard, the training process naturally filters out idiosyncratic hallucinations.
The Evaluation Dilemma
The prevalence of noisy references explains a long-standing paradox in natural language generation: why models that produce highly factual text often score poorly on traditional benchmarks.
Standard automated metrics like ROUGE and BLEU measure simple n-gram overlap between the model's output and the reference text. If a reference summary contains ungrounded fluff or external facts, a perfectly faithful model that refuses to hallucinate will be penalized. It will miss those extrinsic n-grams, resulting in a lower score.
As Meng Cao et al. demonstrate in Learning with Rejection for Abstractive Text Summarization, overlap-based metrics frequently correlate negatively with actual factuality improvements. This misalignment creates an artificial ceiling for model development. To build truly reliable systems, the field has increasingly shifted away from reference-based n-gram matching toward reference-free evaluation metrics—such as natural language inference (NLI) and question-answering frameworks—that judge an output based strictly on its logical entailment from the source document.