Sequence Level Contrastive Learning for Text Summarization

Shusheng Xu, Xingxing Zhang, Yi Wu, Furu Wei

Abstract

Contrastive learning models have achieved great success in unsupervised visual representation learning, which maximize the similarities between feature representations of different views of the same image, while minimize the similarities between feature representations of views of different images. In text summarization, the output summary is a shorter form of the input document and they have similar meanings. In this paper, we propose a contrastive learning model for supervised abstractive text summarization, where we view a document, its gold summary and its model generated summaries as different views of the same mean representation and maximize the similarities between them during training. We improve over a strong sequence-to-sequence text generation model (i.e., BART) on three different summarization datasets. Human evaluation also shows that our model achieves better faithfulness ratings compared to its counterpart without contrastive objectives.

Introduction

Most modern abstractive summarization models, such as BART or T5, are traditionally trained using Negative Log-Likelihood (NLL). This objective functions at a token level, forcing the model to predict the next word in a gold-standard summary. However, NLL is inherently "myopic"; it focuses on local word patterns but does not explicitly reward the model for capturing the global semantic meaning of the document.

Ideally, a document and its summary should be semantically equivalent despite differences in length and vocabulary. Standard training does not guarantee this alignment, which often results in "hallucinations" or summaries that fail to capture the core message. To address this, we introduce SeqCo (Sequence Level Contrastive Learning), a framework that bridges the gap by bringing contrastive learning—a technique that has revolutionized computer vision—into sequence-to-sequence text generation.

Methodology

SeqCo integrates a contrastive learning objective into the fine-tuning stage of a BART model. The core idea is to treat the document ($X$), its gold summary ($Y$), and a model-generated summary ($\hat{Y}$) as different "views" of the same underlying information.

The SeqCo framework treats the document, gold summary, and generated summary as different views of the same semantic representation.

The Contrastive Objective

Unlike many contrastive methods that require explicit "negative" examples, SeqCo utilizes a self-predictive approach inspired by the BYOL (Bootstrap Your Own Latent) method. The goal is to maximize the similarity between the different views. To ensure training stability and prevent "representation collapse," we employ a momentum encoder ($f_\xi$), which is a slow-moving average of the standard encoder ($f_\theta$).

Multi-Head Attention (MHA) Similarity

Rather than comparing single vectors (like a [CLS] token), SeqCo uses a cross-attention mechanism to align sequences. This allows the model to effectively compare a long document to a short summary by focusing on word-to-word interactions. The final loss function is a weighted combination of the standard NLL loss and the contrastive similarity losses between chosen pairs (e.g., Doc-Gold, Doc-Generated, or Gold-Generated).

Experimental Results

We evaluated SeqCo on three standard large-scale benchmarks: CNN/DailyMail (CNNDM), New York Times (NYT), and XSum.

Performance comparison on the CNN/DailyMail dataset. SeqCo variants consistently outperform standard abstractive baselines.

Key Findings

  • State-of-the-Art Performance: SeqCo consistently outperformed the standard BART baseline across all datasets. On CNNDM, the SeqCo ($\lambda_{y-\hat{y}}$) variant achieved a ROUGE-1 score of 45.02, a significant improvement over BART's 44.16.
  • Generated Summaries as Teachers: Interestingly, maximizing similarity between the document and model-generated summaries often yielded better results than using gold-standard summaries alone. Generated summaries are "closer" to the model's current state, providing a more useful signal during the learning process.
  • Improved Faithfulness: Human evaluations confirmed that SeqCo's summaries are more "faithful" (factually consistent with the source) than those produced by standard BART, likely due to the sequence-level semantic alignment.

Discussion and Conclusion

While SeqCo provides significant gains in performance and faithfulness, it does introduce computational overhead. Iteratively generating summaries during training increases training time by approximately 30% per additional contrastive pair. However, we found that using just a single contrastive pair, specifically Gold-Generated ($y-\hat{y}$), provides the best balance of performance and efficiency.

Future work will explore extending SeqCo to multi-lingual summarization tasks, where semantic alignment across different languages is critical. By ensuring that documents and summaries share the same semantic space, SeqCo moves abstractive summarization closer to human-level understanding and factual consistency.

Summary updated

Abstract

Contrastive learning models have achieved great success in unsupervised visual representation learning, which maximize the similarities between feature representations of different views of the same image, while minimize the similarities between feature representations of views of different images. In text summarization, the output summary is a shorter form of the input document and they have similar meanings. In this paper, we propose a contrastive learning model for supervised abstractive text summarization, where we view a document, its gold summary and its model generated summaries as different views of the same mean representation and maximize the similarities between them during training. We improve over a strong sequence-to-sequence text generation model (i.e., BART) on three different summarization datasets. Human evaluation also shows that our model achieves better faithfulness ratings compared to its counterpart without contrastive objectives. We release our code at https://github.com/xssstory/SeqCo.
BibTeX
@inproceedings{xu2022sequence,
  author = {Shusheng Xu and Xingxing Zhang and Yi Wu and Furu Wei},
  title = {{Sequence Level Contrastive Learning for Text Summarization}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  url = {https://openreview.net/forum?id=Id5p86Xgje},
}
Figure