Noised Consistency Training for Text Summarization

Junnan Liu, Qianren Mao, Bang Liu, Hao Peng, Hongdong Zhu, Jianxin LiarXiv 2021 · OpenReview

TL;DR

This paper introduces a semi-supervised learning approach called "Noised Consistency Training" to improve abstractive text summarization, especially when labeled training data is scarce. By leveraging large amounts of unlabeled text and applying data augmentation techniques to introduce "noise," the model learns to produce consistent summaries, achieving competitive performance without needing a fully labeled dataset.

What Problem Does This Paper Address?

Training high-performing neural abstractive summarization models typically requires vast amounts of meticulously labeled data (source text paired with its summary). However, creating these labeled datasets is expensive and time-consuming, limiting the practical application of these powerful models. This paper tackles the challenge of building effective summarization systems when faced with insufficient labeled data, aiming to bridge the gap between research potential and real-world deployment.

Methods

The core of this work is Noised Consistency Training, a semi-supervised learning technique adapted for text summarization. The approach involves:

  1. Leveraging Unlabeled Data: The framework utilizes a large corpus of unlabeled text in addition to a smaller set of labeled data.
  2. Data Augmentation: To introduce "noise" and encourage robustness, two data augmentation methods are employed on the unlabeled text:
  3. Back-translation: Translating a document to another language and then back to the original language. This generates paraphrased versions that retain the original meaning but offer different phrasing.
  4. TF-IDF Word Replacement: Replacing less important words (low TF-IDF scores) with other non-informative words, while preserving key terms.
  5. Consistency Learning: The model is trained to produce similar summary distributions for both the original unlabeled text and its augmented, "noised" versions. This consistency regularization forces the model to be invariant to minor variations in the input, making it more robust.
  6. Joint Optimization: The training process combines a standard supervised loss (using labeled data) with an unsupervised consistency loss (using unlabeled and augmented unlabeled data).
  7. Backbone Model: The authors use a BERT2BERT model, which leverages BERT for both encoding the input text and decoding the summary, benefiting from BERT's strong language understanding capabilities.

Datasets

The experiments were conducted on two widely used datasets for text summarization:

  • CNN/DailyMail: A large dataset of news articles and their summaries.
  • BBC XSum: A dataset known for its highly abstractive, single-sentence summaries.

The authors created subsets of these datasets by splitting the full datasets into labeled and unlabeled portions. They experimented with different proportions of labeled data (30%, 50%, 70%) paired with the remaining unlabeled data.

Key Insights and Ideas

  • Consistency is Key for Robustness: The fundamental idea is that a good summarization model should be robust to small variations in the input text. By forcing the model to produce consistent outputs for slightly altered inputs, it learns to focus on the core semantic meaning.
  • Data Augmentation as a "Noise Injector": Instead of simple noise injection, advanced techniques like back-translation prove more effective. Back-translation not only adds diversity but also helps maintain the global semantics of the original text, which is crucial for summarization.
  • Semi-Supervised Learning for Summarization is Viable: The paper demonstrates that semi-supervised learning, specifically consistency training, can significantly boost performance even with limited labeled data, making it a practical approach for real-world summarization tasks.
  • Unlabeled Data is Highly Valuable: The results strongly suggest that a large amount of unlabeled data, when used effectively through consistency training, can compensate for a lack of labeled data, leading to performance comparable to models trained on much larger labeled sets.

Key Results

  • Competitive Performance with Less Data: On the CNN/DailyMail dataset, the proposed consistency training framework achieved performance comparable to models trained on the full labeled dataset, even when using only 30% of the labeled data. For instance, with 30% labeled data, consistency training improved ROUGE-1 scores from 37.85 to 39.06.
  • Significant Gains on Limited Data: When using only 30% labeled data, consistency training provided substantial performance gains compared to a supervised baseline using the same limited labeled data (e.g., +1.58 on ROUGE-2).
  • Effectiveness on Abstractive Summaries: The method also showed promise on the highly abstractive BBC XSum dataset, with consistency training achieving similar results to the full dataset baseline even with 70% labeled data.
  • Back-translation Outperforms TF-IDF: Back-translation was found to be a superior data augmentation method compared to TF-IDF word replacement, leading to better ROUGE scores on average.
  • Batch Size Matters: For unlabeled data, a batch size of 32 provided optimal convergence speed and performance, outperforming batch sizes of 16 and 64.

Limitations and Future Work

  • Limitations: The paper acknowledges that while consistency training with unlabeled data improves performance, it doesn't fully surpass models trained on the complete labeled dataset in all scenarios. The effectiveness of data augmentation methods might vary across different domains.
  • Future Work: The authors suggest migrating their consistency training framework to other natural language generation tasks, such as question answering and dialogue generation, to explore its broader applicability.

Summary updated

Abstract

Neural abstractive summarization methods often require large quantities of labeled training data. However, labeling large amounts of summarization data is often prohibitive due to time, financial, and expertise constraints, which has limited the usefulness of summarization systems to practical applications. In this paper, we argue that this limitation can be overcome by a semi-supervised approach: consistency training which is to leverage large amounts of unlabeled data to improve the performance of supervised learning over a small corpus. The consistency regularization semi-supervised learning can regularize model predictions to be invariant to small noise applied to input articles. By adding noised unlabeled corpus to help regularize consistency training, this framework obtains comparative performance without using the full dataset. In particular, we have verified that leveraging large amounts of unlabeled data decently improves the performance of supervised learning over an insufficient labeled dataset.
BibTeX
@misc{liu2021noised,
  author = {Junnan Liu and Qianren Mao and Bang Liu and Hao Peng and Hongdong Zhu and Jianxin Li},
  title = {{Noised Consistency Training for Text Summarization}},
  year = {2021},
  eprint = {2105.13635},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2105.13635},
}
Figure