Improving the Robustness of Summarization Systems with Dual Augmentation

Xiuying Chen, Guodong Long, Chongyang Tao, Mingzhe Li, Xin Gao, Chengqi Zhang, Xiangliang Zhang

TL;DR

State-of-the-art summarization models are surprisingly fragile, often failing when faced with simple typos or synonym substitutions. This paper introduces DASum, a dual-augmentation framework that bolsters model robustness by diversifying training data in both the discrete input space and the continuous latent (hidden) space.

What Problem Does This Paper Address?

While humans can effortlessly summarize a text despite typos or varied wording, neural summarization models are notoriously brittle. A single word change—like swapping "attorney" for "barrister"—can lead a model to hallucinate or omit critical information.

The authors identify two primary "brittleness factors":

  1. Vocabulary Sensitivity: Models struggle with infrequent or "long-tail" words introduced by noise or diverse phrasing.
  2. Latent Instability: Small input perturbations amplify as they flow through the model's decoder, leading to wildly different internal representations (hidden states) compared to clean text.

Methods

The authors propose a two-pronged defense strategy called DASum (Dual Augmentation for Summarization), supported by a new attack method:

  • SummAttacker: To test and train for robustness, the authors developed an efficient adversarial attack. It identifies influential words using the model's own attention scores and replaces them with synonyms that are both fluent (according to a language model) and "adversarial" (moving the model's internal state in the opposite direction of the optimization gradient).
  • Input Space Augmentation: The encoder is trained on a mix of clean text, noisy text (typos), and adversarial samples generated by SummAttacker. This helps the model better understand infrequent words and varied expressions.
  • Latent Semantic Space Augmentation: To stabilize the decoder, the authors use manifold softmixing. During training, they create "virtual" hidden states by interpolating between the hidden states of clean and adversarial inputs at randomly selected decoder layers. This forces the model to map diverse internal representations to the same correct summary.
  • Dual Augmented Consistency: A Kullback-Leibler (KL) divergence loss is added to ensure the model's output probability distributions remain consistent across original and augmented versions of the same input.
Figure 3: (a) Illustration of training examples sampled from vicinity distributions that could cover variants of literal expression under the same meaning. (b) The architecture of our dual data augmentation approach.

Datasets

The researchers evaluated their approach on two standard benchmarks that offer different document lengths:

  • Gigaword: Short documents (avg. 70 words), typically used for headline generation.
  • CNN/DailyMail (CNN/DM): Longer documents (avg. 700 words) requiring more complex summarization.

For noise evaluation, they used a look-up table of naturally occurring human errors (typos and misspellings) to perturb the test sets.

Key Insights and Ideas

  • The "Infrequent Word" Trap: Analysis revealed that replacement words in successful attacks were, on average, four times less frequent than the original words. This confirms that models rely heavily on high-frequency patterns and "break" when forced to process rarer synonyms.
  • Error Amplification in Decoders: By visualizing hidden states with t-SNE, the authors showed that the "distance" between clean and attacked representations grows as information moves from the first to the last decoder layer. DASum effectively shrinks this gap, as seen in Figure 2.
  • Short Documents are More Vulnerable: Models trained on Gigaword were more susceptible to attacks than those on CNN/DM. In shorter texts, each word carries more relative "weight," meaning a single perturbation has a higher chance of distorting the entire summary.
Figure 2: t-SNE visualization of the hidden states in BART and our DASum, when taking original and at- tacked inputs. E is the average Euclidean distance of paired original and attacked states before using t-SNE.

Key Results

DASum consistently outperformed strong baselines like BART, ProphetNet, and R3F across all metrics (ROUGE, BERTScore, and factual consistency via QuestEval).

  • Robustness to Noise: On the Gigaword dataset, while standard BART and other augmentation methods (like SSTIA) saw sharp performance drops as noise increased, DASum maintained a significantly flatter curve (see Figure 5).
  • Adversarial Defense: Under a 1% token attack on Gigaword, DASum achieved a ROUGE-L of 35.31, compared to BART's 32.52. On CNN/DM (5% attack), DASum reached 39.08 ROUGE-L, significantly higher than the 33.36 achieved by the baseline BART.
  • Ablation Success: Removing the latent space augmentation ($L_d$) or the input space augmentation ($L_e$) led to measurable performance drops, proving that both "halves" of the dual approach are necessary for maximum robustness.
Figure 5: The impact of noise on the performance of summarization models on Gigaword. While SSTIA and BART show significant drops in all metrics, our DASum has a robust performance. The noise here consists of multiple human errors (typos, misspellings, etc.)

Limitations and Future Work

  • Word-Level Focus: The study primarily addresses word-level changes (synonyms, typos). The authors note that future work should explore robustness against sentence-level or document-level perturbations (e.g., reordering or paraphrasing entire sections).
  • Computational Cost: Generating the augmented dataset via SummAttacker requires additional computational resources during the training phase, which might be a bottleneck for extremely large-scale datasets.
  • Architecture Specificity: While tested on BART, the manifold softmixing approach is tied to Transformer-like architectures; its effectiveness on other emerging architectures remains to be seen.

Summary updated

Abstract

A robust summarization system should be able to capture the gist of the document, regardless of the specific word choices or noise in the input. In this work, we first explore the summarization models' robustness against perturbations including word-level synonym substitution and noise. To create semantic-consistent substitutes, we propose a SummAttacker, which is an efficient approach to generating adversarial samples based on language models. Experimental results show that state-of-the-art summarization models have a significant decrease in performance on adversarial and noisy test sets. Next, we analyze the vulnerability of the summarization systems and explore improving the robustness by data augmentation. Specifically, the first brittleness factor we found is the poor understanding of infrequent words in the input. Correspondingly, we feed the encoder with more diverse cases created by SummAttacker in the input space. The other factor is in the latent space, where the attacked inputs bring more variations to the hidden states. Hence, we construct adversarial decoder input and devise manifold softmixing operation in hidden space to introduce more diversity. Experimental results on Gigaword and CNN/DM datasets demonstrate that our approach achieves significant improvements over strong baselines and exhibits higher robustness on noisy, attacked, and clean datasets.
BibTeX
@inproceedings{chen2023improving,
  author = {Xiuying Chen and Guodong Long and Chongyang Tao and Mingzhe Li and Xin Gao and Chengqi Zhang and Xiangliang Zhang},
  title = {{Improving the Robustness of Summarization Systems with Dual Augmentation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.378},
}
Figure