RewardsOfSum: Exploring Reinforcement Learning Rewards for Summarisation

Jacob Parnell, Inigo Jauregi Unanue, Massimo PiccardiACL 2021

TL;DR

This paper tackles the challenge of training abstractive summarization models by exploring novel reinforcement learning (RL) reward functions beyond the standard token-level negative log-likelihood (NLL). The authors introduce two new reward functions, RwB-Hinge and RISK, which aim to better align training objectives with sequence-level evaluation metrics like ROUGE, leading to improved summary quality and diversity.

What Problem Does This Paper Address?

Abstractive summarization models, which generate novel sentences rather than just copying from the source, typically use negative log-likelihood (NLL) as their training objective. While effective, NLL has limitations: it treats each token independently, ignoring synonyms and other valid phrasing, and it optimizes a token-level objective that doesn't directly align with sequence-level evaluation metrics like ROUGE. This mismatch can lead to summaries that are technically "correct" according to NLL but don't perform as well on human-centric evaluations. Reinforcement learning (RL) offers a way to optimize directly for these sequence-level metrics, but choosing an effective reward function is crucial and has been an area of active research. The paper aims to address this by proposing and evaluating new RL reward functions that are more attuned to the nuances of summarization.

Methods

The paper proposes two novel reward functions for abstractive summarization within a reinforcement learning framework:

  • RwB-Hinge: This method modifies the standard REINFORCE algorithm with a baseline. It incorporates a hinge loss that selectively applies model gradients, focusing updates on predictions that are considered "good" based on their reward score relative to a baseline prediction. This aims to prevent inappropriately penalizing acceptable predictions and guide training more effectively.
  • RISK (Expected Risk Minimization): This approach leverages a small pool of strong candidate summaries (sampled or predicted) to inform the reward. Instead of relying on a single baseline, it uses multiple diverse candidates, allowing the model to learn from a richer set of potential outputs and avoid being overly biased by a single "best" prediction.

Both methods are integrated into a mixed training objective, combining the RL reward with the standard NLL loss to maintain a balance and prevent the model from deviating too far from the ground truth.

Datasets

The authors evaluate their proposed methods on nine diverse summarization datasets, categorized by the length of their reference summaries:

  • Short: AESLC, Gigaword, XSum
  • Medium: CNN/DM, Reddit-TIFU, Newsroom
  • Long: Pubmed, ArXiv, Billsum

These datasets were chosen to provide a broad range of summarization tasks and reference lengths, allowing for a comprehensive evaluation of the proposed reward functions.

Key Insights and Ideas

  • NLL's limitations are significant: The paper strongly emphasizes that optimizing for token-level NLL doesn't directly translate to better performance on sequence-level metrics like ROUGE, which are more representative of summary quality.
  • Reward function design is critical for RL in summarization: Simply applying standard RL techniques isn't enough; the reward signal needs to be carefully crafted to guide the model towards generating high-quality, diverse summaries.
  • Diversity in sampling aids learning: The RISK method's success highlights the benefit of exposing the model to multiple valid summary candidates during training, rather than just the single best prediction or the ground truth.
  • Hinge loss can refine gradient updates: The RwB-Hinge approach suggests that selectively applying gradients based on a reward threshold can lead to more stable and effective training.
  • RL can improve long summary generation: The results indicate that the proposed RL methods can help overcome the tendency of NLL models to truncate longer summaries, leading to better ROUGE scores for longer outputs.

Key Results

The proposed RwB-Hinge and RISK methods consistently outperformed the standard NLL baseline across the nine datasets, for both few-shot and full-data training scenarios.

  • Consistent Improvement: Across most datasets, both RwB-Hinge and RISK showed statistically significant improvements in ROUGE scores (ROUGE-1, ROUGE-2, and ROUGE-L) compared to the NLL baseline.
  • N-gram Novelty: The RL-trained models generated summaries with higher n-gram novelty, indicating greater abstractiveness and diversity compared to NLL-trained models. For example, tri-gram novelty increased by up to 13.8 percentage points on CNN/DM.
  • Longer Summaries Benefit: The RL approaches showed a notable improvement in ROUGE-L scores for longer summaries, with gains of up to 6.2 points for summaries over 100 tokens, suggesting they help mitigate the tendency of NLL models to produce shorter outputs.
  • Dataset-Specific Performance: Table 7 suggests that RISK methods performed best on short/medium datasets, while RwB-Hinge showed an advantage on longer datasets.

Limitations and Future Work

  • ROUGE as a Reward: The authors acknowledge that while ROUGE is a common evaluation metric, directly optimizing for it as a reward in RL doesn't always translate to improved performance. They suggest that this might be due to the inherent limitations of ROUGE itself.
  • Future Work: The paper proposes exploring other contemporary score functions, such as BERTScore, as potential rewards to build more effective summarization models. They also plan to investigate other sampling strategies and RL algorithms.

Summary updated

BibTeX
@inproceedings{parnell2021rewardsofsum,
  author = {Jacob Parnell and Inigo Jauregi Unanue and Massimo Piccardi},
  title = {{RewardsOfSum: Exploring Reinforcement Learning Rewards for Summarisation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure