Differentiable <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" altimg="si206.svg" display="inline" id="d1e1382"><mml:mi>N</mml:mi></mml:math>-gram objective on abstractive summarization

Yunqi Zhu, Xuebing Yang, Yuanyuan Wu, Mingjin Zhu, Wensheng ZhangExpert 2022 · DOI

TL;DR

This paper introduces novel "differentiable N-gram objectives" to bridge the gap between how abstractive summarization models are trained (using cross-entropy loss) and how they are evaluated (using metrics like ROUGE that focus on N-gram overlap). By jointly optimizing these new objectives with cross-entropy, the authors achieve improved summarization quality, outperforming existing methods on standard benchmarks.

What Problem Does This Paper Address?

Automatic text summarization, particularly abstractive summarization (where the model generates new sentences rather than just copying existing ones), is a challenging task. A key issue lies in the mismatch between the training objective and the evaluation metric. Most abstractive summarization models are trained using cross-entropy loss, which focuses on predicting the next token correctly based on the ground truth. However, evaluation metrics like ROUGE, which are widely used to assess summary quality, rely on measuring the overlap of N-grams (sequences of N words) between the generated summary and a reference summary. This discrepancy can lead to models that perform poorly on evaluation metrics despite achieving low training loss. The paper aims to alleviate this "exposure bias" by creating training objectives that more closely align with N-gram-based evaluation.

Methods

The core contribution of this paper is the proposal of two novel differentiable N-gram objectives:

  • Differentiable N-gram Rewards: This objective focuses on position-related N-gram matches. It aims to maximize the probability of matched N-grams between the generated and reference summaries, but only if the N-gram appears at the same position in both sequences. This is a more constrained form of matching.
  • Differentiable N-gram Matches: This objective is position-unrelated. It maximizes the probability of matched N-grams regardless of their position in the sequences. This aligns more closely with how ROUGE scores are calculated.

Both objectives are designed to be differentiable, allowing them to be directly incorporated into the training process using gradient-based optimization. Crucially, these new objectives do not "ceil" the number of matched N-grams based on the reference count and weight all matched sub-sequences equally. This allows the model to be rewarded for multiple occurrences of the same N-gram, even if it exceeds the count in the reference. The authors jointly optimize these N-gram objectives with the standard cross-entropy loss.

Datasets

The paper uses the following standard datasets for abstractive summarization:

  • CNN/DailyMail: A widely used dataset containing news articles and their corresponding highlights.
  • XSum: Another popular dataset consisting of BBC news articles paired with single-sentence summaries, requiring a higher degree of text compression.

Key Insights and Ideas

  • Bridging the Training-Evaluation Gap: The fundamental insight is that aligning training objectives with evaluation metrics is crucial for improving performance on those metrics. Cross-entropy is a good objective for token-level prediction, but N-gram-based metrics require a different focus.
  • Differentiable N-gram Objectives: Making N-gram matching differentiable is key to enabling its use as a direct training objective, overcoming the limitations of non-differentiable metrics.
  • No Ceiling on N-grams: Removing the constraint of matching only up to the count in the reference allows the model to be more robustly trained on N-gram overlap, especially for abstractive summaries where paraphrasing is common.
  • Position-Related vs. Position-Unrelated: The paper explores two variations of N-gram matching, suggesting that the choice might depend on the specific task or dataset, with position-unrelated matching being closer to ROUGE.

Key Results

The proposed differentiable N-gram objectives, when jointly trained with cross-entropy on the BART-base model, demonstrated significant improvements on both CNN/DM and XSum datasets.

  • ROUGE Scores: On CNN/DM, the (2,3,4)-gram rewards objective achieved an increase of 0.39/0.32/0.48 in ROUGE-1/R-2/R-L scores compared to the baseline BART-base. The (2,3,4)-gram matches objective also showed improvements, with R-L scores reaching the level of BART-large. For XSum, optimizing with 2-gram objectives yielded the best results, with increases of 0.30/0.25/0.45 in R-1/R-2/R-L.
  • BERTScore: The proposed methods generally showed slight improvements in BERTScore, indicating better contextual similarity.
  • Factual Consistency: The SummaCCONV metric showed that the proposed objectives achieved greater factual consistency than competitors on CNN/DM.
  • Efficiency: While the N-gram rewards objective was noted to be more time-consuming, the authors suggest that optimizations could improve throughput.

Limitations and Future Work

  • Hyperparameter Tuning: The number of N-gram objectives to include (e.g., 2-gram, 3-gram, 4-gram) is a hyperparameter that needs to be tuned for different datasets.
  • Dataset Domain: The current study focused on news summarization datasets. Further research is needed to apply these objectives to other domains and pre-trained models.
  • Computational Cost: The N-gram rewards objective can be computationally intensive, though potential optimizations are mentioned.
  • Future Work: The authors suggest exploring these objectives on other sequence-to-sequence tasks and investigating their application to fact-aware summarization models. They also highlight the potential for further code optimization to improve training speed.

Summary updated

BibTeX
@inproceedings{zhu2022differentiable,
  author = {Yunqi Zhu and Xuebing Yang and Yuanyuan Wu and Mingjin Zhu and Wensheng Zhang},
  title = {{Differentiable <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" altimg="si206.svg" display="inline" id="d1e1382"><mml:mi>N</mml:mi></mml:math>-gram objective on abstractive summarization}},
  booktitle = {Expert},
  year = {2022},
  doi = {10.1016/j.eswa.2022.119367},
}
Figure