Abstract
Contrastive learning models have achieved great success in unsupervised visual representation learning, which maximize the similarities between feature representations of different views of the same image, while minimize the similarities between feature representations of views of different images. In text summarization, the output summary is a shorter form of the input document and they have similar meanings. In this paper, we propose a contrastive learning model for supervised abstractive text summarization, where we view a document, its gold summary and its model generated summaries as different views of the same mean representation and maximize the similarities between them during training. We improve over a strong sequence-to-sequence text generation model (i.e., BART) on three different summarization datasets. Human evaluation also shows that our model achieves better faithfulness ratings compared to its counterpart without contrastive objectives.
Introduction
Most modern abstractive summarization models, such as BART or T5, are trained using Negative Log-Likelihood (NLL). This objective treats summarization as a token-level next-word prediction task. However, NLL has significant limitations: it ignores the global semantic alignment between the entire document and the summary, and it suffers from "exposure bias" because the model is only trained on human-written "gold" summaries, not its own generated outputs. To address these gaps, we propose SeqCo (Sequence Level Contrastive Learning), a framework that aligns the semantic representations of documents and summaries in a shared vector space.
Methodology
SeqCo adapts contrastive learning—a technique highly successful in computer vision—to the domain of Natural Language Processing. The core idea is to treat the input document ($X$), the gold summary ($Y$), and the model-generated summary ($\hat{Y}$) as three different "views" of the same semantic information.
Semantic Alignment and Momentum Encoding
To compute similarity between sequences of different lengths (e.g., a long document vs. a short summary), SeqCo utilizes Multi-Head Attention (MHA). This allows the model to capture nuanced word-to-word interactions rather than relying on simple pooling methods like [CLS] tokens. To prevent "representation collapse"—where the model maps all inputs to a single constant vector—SeqCo employs a momentum encoder. In this setup, one side of the comparison is updated via standard gradients, while the other is a slow-moving average of the model’s parameters ($f_\xi$).
Training Objective
The final loss function is a hybrid that maintains the traditional NLL loss for grammatical correctness while adding a contrastive loss term. This forces the model to minimize the distance between document and summary representations, implicitly teaching the encoder to prioritize salient information and ignore "noise."
Experimental Results
We evaluated SeqCo on three benchmark datasets: CNN/DailyMail (CNNDM), New York Times (NYT), and XSum. SeqCo was built on top of the BART-Large architecture and demonstrated consistent improvements across all metrics.
As shown in the results for CNN/DailyMail, SeqCo outperformed the standard BART baseline. Notably, the variant contrasting the document with the model's own generated summaries ($\lambda_{y-\hat{y}}$) often yielded the best results, achieving a ROUGE-L score of 41.75. This suggests that generated summaries help the model learn to correct its own typical errors.
Key Insights
- Faithfulness: Human evaluators rated SeqCo's summaries as more factually accurate relative to the source than those produced by standard BART.
- Abstractiveness: The model maintained high levels of abstractiveness, successfully rephrasing content rather than simply extracting sentences.
- Efficiency: While the contrastive objective increases training time by approximately 30%, it has no impact on inference speed, making it a viable enhancement for production models.
Conclusion
SeqCo demonstrates that sequence-level semantic alignment is a powerful tool for improving abstractive summarization. By treating documents and summaries as different views of the same information, the model achieves better ROUGE scores and higher factual faithfulness. Future work will explore applying this framework to multi-lingual and cross-lingual summarization tasks.
Summary updated