R-TeaFor: Regularized Teacher-Forcing for Abstractive Summarization

Guan-Yu Lin, Pu-Jen ChengEMNLP 2022 · OpenReview

Introduction

Abstractive summarization models often struggle with a phenomenon known as "exposure bias." R-TeaFor (Regularized Teacher-Forcing) is a novel training framework designed to bridge the gap between training and inference by enforcing consistency between ground-truth and noisy, augmented target sequences.

The Challenge: Exposure Bias

Most sequence generation models are trained using Teacher-Forcing, where the model is provided with the correct previous word from the ground-truth data to predict the next word. However, during inference, the ground truth is unavailable, and the model must rely on its own previously generated—and potentially erroneous—words. This discrepancy leads to exposure bias: small errors early in a sequence can accumulate, causing the model to drift into "uncharted territory" where performance degrades rapidly.

Methodology: Regularized Teacher-Forcing

The authors propose Regularized Teacher-Forcing (R-TeaFor), which introduces a consistency constraint into the standard training loop.

The R-TeaFor training framework architecture showing dual-stream processing and consistency regularization.

The framework operates through several key steps:

  1. Dual-Stream Processing: In each training step, the model processes the source sequence twice: once with the original ground-truth target sequence ($t$) and once with an augmented target sequence ($t'$).
  2. Data Augmentation via Masking: The augmented sequence $t'$ is created by randomly replacing tokens in the ground truth with a <mask\> token (with probability $\beta$). This simulates the uncertainty and potential errors the model encounters during real-world inference.
  3. Multi-Part Objective Function: R-TeaFor optimizes a combined loss:
  4. Negative Log-Likelihood (NLL): The standard cross-entropy loss applied to both original and augmented sequences.
  5. Symmetric KL-Divergence: This acts as a regularizer, penalizing the model if the probability distributions it predicts for the next word differ significantly between the ground-truth input and the masked input.
  6. Consistency as a Guide: By training on both streams simultaneously, the ground-truth sequence serves as a stable anchor, preventing the model from drifting toward sub-optimal, noisy predictions.

Experimental Setup and Datasets

The researchers evaluated R-TeaFor on two primary benchmarks for abstractive summarization:

  • CNN/DailyMail: A dataset of news articles with multi-sentence summaries, testing the model's ability to compress long-form content.
  • XSum (Extreme Summarization): A more challenging dataset requiring highly abstractive, single-sentence summaries.

Key Insights and Ideas

A critical finding of the research is that pairwise relationships between clean and noisy data are more effective than simply adding noise. Forcing the model to remain consistent relative to the ground truth makes it significantly more robust.

Cosine similarity distribution between ground-truth and self-generated inputs across different training methods.

As shown in the analysis of prediction similarity, R-TeaFor achieves much higher stability than standard models. Specifically, 85.5% of R-TeaFor's predictions showed a cosine similarity higher than 0.75 between ground-truth and self-generated inputs. This indicates that the model's output remains stable even when it encounters the types of perturbations common during inference. Furthermore, while all models experience a drop in "Decoding Accuracy" as sequences grow longer, R-TeaFor maintains a significantly higher accuracy and a smaller "accuracy gap" compared to standard BART or PEGASUS models.

Key Results

R-TeaFor consistently outperformed state-of-the-art baselines across both datasets:

  • CNN/DailyMail: BART + R-TeaFor achieved a ROUGE-1 score of 45.52, surpassing standard BART (44.16).
  • XSum: PEGASUS + R-TeaFor reached a ROUGE-1 of 47.90, improving upon the base PEGASUS (47.21).

The method's success across different architectures (BART and PEGASUS) suggests it is a versatile "plug-and-play" strategy for various Transformer-based generators.

Limitations and Future Work

While effective, R-TeaFor comes with increased computational costs. Training time effectively doubles because the model must process the target sequence twice per step, typically requiring around 48 hours on a Tesla V100 GPU for standard runs. Additionally, the regularization effect is most pronounced in the earlier stages of decoding; its impact on extremely long sequences remains a subject for future study. Beyond summarization, the authors suggest R-TeaFor holds promise for other sequence-to-sequence tasks such as machine translation and question answering.

Summary updated

Abstract

Teacher-forcing is widely used in training sequence generation models to improve sampling efficiency and to stabilize training. However, teacher-forcing is vulnerable to the exposure bias problem. Previous works have attempted to address exposure bias by modifying the training data to simulate model-generated results. Nevertheless, they do not consider the pairwise relationship between the original training data and the modified ones, which provides more information during training. Hence, we propose Regularized Teacher-Forcing (R-TeaFor) to utilize this relationship for better regularization. Empirically, our experiments show that R-TeaFor outperforms previous summarization state-of-the-art models, and the results can be generalized to different pre-trained models.
BibTeX
@inproceedings{lin2022r,
  author = {Guan-Yu Lin and Pu-Jen Cheng},
  title = {{R-TeaFor: Regularized Teacher-Forcing for Abstractive Summarization}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=W7eNuPDzLF},
}
Figure