Introduction
Abstractive summarization models like BART and PEGASUS have achieved remarkable fluency, yet they frequently suffer from "hallucinations"—generating content that is factually inconsistent with the source text. This paper introduces CLIFF (Contrastive Learning for Improving Faithfulness and Factuality), a framework designed to mitigate these errors by training models to distinguish between correct references and automatically generated erroneous samples.
The Hallucination Problem
Standard training using Maximum Likelihood Estimation (MLE) encourages models to mimic reference summaries but fails to penalize common factual errors. Consequently, models often produce summaries that are fluent but incorrect. Errors are generally categorized into intrinsic (misinterpreting source information) and extrinsic (introducing information not present in the source).
As shown in the figure above, extrinsic errors are particularly prevalent in abstractive datasets like XSum, where models often generate information beyond the provided context.
The CLIFF Framework
CLIFF addresses this by augmenting the standard cross-entropy loss with a contrastive learning objective. The model is trained to pull its internal representations closer to a "positive" reference summary while pushing them away from "negative" samples containing factual errors.
Generating Negative Samples
The success of CLIFF relies on high-quality negative samples that mimic real-world failure modes:
- Entity Swap: Swapping named entities (e.g., names, locations) with others of the same type from the source.
- Mask-and-fill: Using BART to fill masked entities in a reference, often leading to fluent but ungrounded hallucinations.
- Source-conditioned Regeneration: Generating continuations based on the source that are plausible but factually incorrect.
- SysLowCon (System Low Confidence): Utilizing the model's own low-confidence outputs for proper nouns or numbers as organic negative examples.
Results and Analysis
CLIFF was evaluated on the XSum and CNN/Daily Mail benchmarks. The results demonstrate that CLIFF significantly reduces both intrinsic and extrinsic errors compared to standard cross-entropy and other baselines.
Key findings include:
- Error Reduction: CLIFF is especially effective at curbing extrinsic hallucinations, which are typically the hardest to address.
- Confidence as a Signal: The authors observed that models tend to have lower token-level probability when hallucinating extrinsic information, a signal leveraged by the SysLowCon strategy.
- Human Preference: In human evaluations on XSum, CLIFF summaries were preferred for factual consistency 31.3% of the time over the baseline, while being rated worse only 7.0% of the time.
Conclusion
CLIFF provides a robust framework for improving the factuality of abstractive summarizers without requiring additional parameters or sacrificing abstractive quality. By explicitly teaching models to recognize and avoid common errors through contrastive learning, CLIFF moves closer to generating summaries that are both fluent and trustworthy.
Summary updated