Abstract
We study generating abstractive summaries that are faithful and factually consistent with the given articles. A novel contrastive learning formulation is presented, which leverages both reference summaries, as positive training data, and automatically generated erroneous summaries, as negative training data, to train summarization systems that are better at distinguishing between them. We further design four types of strategies for creating negative samples, to resemble errors made commonly by two state-of-the-art models, BART and PEGASUS, found in our new human annotations of summary errors. Experiments on XSum and CNN/Daily Mail show that our contrastive learning framework is robust across datasets and models. It consistently produces more factual summaries than strong comparisons with post error correction, entailment-based reranking, and unlikelihood training, according to QA-based factuality evaluation. Human judges echo the observation and find that our model summaries correct more errors.
Introduction
State-of-the-art abstractive summarization models like BART and PEGASUS are incredibly fluent, but they frequently generate "hallucinations"—content that is either factually inconsistent with the source (intrinsic errors) or includes unverifiable information not present in the source (extrinsic errors). The root of the problem lies in the standard training objective: Maximum Likelihood Estimation (MLE). MLE encourages models to mimic reference summaries but doesn't explicitly teach them to avoid common pitfalls or distinguish between a factual statement and a plausible-sounding lie.
The CLIFF Framework
The authors introduce CLIFF (Contrastive Learning for Improving Faithfulness and Factuality). The core idea is to use a contrastive loss function that pulls the model's representations of "positive" (faithful) summaries closer together while pushing "negative" (erroneous) summaries further away.
Negative Sample Construction
The "secret sauce" of CLIFF is how it constructs these negative samples to mimic real-world model errors:
- Entity Swap (SWAPENT): Swaps named entities in a reference summary with other entities of the same type found in the source article.
- Mask-and-fill (MASKENT/MASKREL): Replaces entities or relations with
[MASK]tokens and lets a pre-trained BART model fill them in. - Source-conditioned Regeneration (REGENENT/REGENREL): Uses the beginning of a reference summary as a prompt and lets the model finish it based on the source article.
- System Generation (SYSLOWCON): This "hard negative" strategy leverages the finding that models are often less confident (lower token probability) when they are about to make an error.
Key Insights
- Confidence as a Compass: As shown in the figure above, tokens in extrinsic error spans have significantly lower generation probabilities than correct tokens or even "world knowledge" tokens. This observation justifies the use of low-confidence generations as hard negative samples.
- Diversity of Negatives Matters: No single strategy for creating negative samples is a silver bullet. Combining different types of errors (swaps, regenerations, and low-confidence outputs) provides the most robust training signal.
- Representation Choice: Averaging the decoder's hidden states for all tokens and passing them through a Multi-Layer Perceptron (MLP) projection worked best for the contrastive loss, capturing global context effectively.
Results and Conclusion
CLIFF consistently outperformed strong baselines—including unlikelihood training and entailment-based reranking—on QuestEval, a state-of-the-art QA-based factuality metric. Human judges confirmed that CLIFF summaries were more factual and informative. On the challenging XSum dataset, CLIFF's "low confidence" variant was rated better than the standard cross-entropy baseline in over 31% of cases for factuality.
While CLIFF makes significant strides, it doesn't eliminate hallucinations entirely. Future work should focus on more complex relation-based errors and exploring "cross-model" negative samples to further improve generalization.
Summary updated