Good Rewards Are All You Need: Reward Learning for Efficient Reinforcement Learning in Document Summarisation

Yingfang Gao, Christian M. Meyer, Mohsen Mesgar, Iryna GurevychTUbilio 2019 · OpenAlex

Abstract

Extractive document summarization is often framed as a reinforcement learning (RL) problem to optimize non-differentiable metrics like ROUGE. However, standard RL approaches are either computationally expensive (cross-input RL) or rely on suboptimal handcrafted rewards (input-specific RL). We present RELIS (REward Learning for Input-Specific RL), a framework that decouples reward learning from policy search. By training an approximated reward oracle using Learning-to-Rank (L2R) during a brief training phase, RELIS enables rapid, high-quality RL at test time. Our approach achieves a two-order-of-magnitude reduction in training time while matching or exceeding the performance of state-of-the-art neural summarization models.

Introduction

The goal of extractive summarization is to select a subset of sentences that best represent the source text. Framing this as a sequential decision-making process allows for the direct optimization of ROUGE scores via Reinforcement Learning. However, practitioners face a dilemma. Cross-input RL models, which learn a global policy, suffer from massive search spaces and sparse rewards, leading to slow convergence. Conversely, input-specific RL models are faster but depend on handcrafted reward functions that often fail to capture the nuances of human-quality summaries.

The RELIS Framework

RELIS addresses these limitations by learning a high-quality reward function that can be deployed for fast, document-specific optimization at test time.

The RELIS framework compared to traditional Cross-input and Input-specific RL paradigms.

The framework consists of two distinct phases:

  1. Training Phase (Reward Learning): Instead of training a policy, RELIS trains an approximated reward oracle. Using Learning-to-Rank (L2R) algorithms, it learns a utility function ($U$) that predicts summary quality based on "weak supervisions"—either numeric ROUGE scores (pointwise) or pairwise preferences.
  2. Test Phase (Input-Specific RL): For a new document, a fresh RL agent is initialized. This agent uses the pre-trained approximated reward oracle to quickly learn a policy tailored to that specific input.

Theoretical analysis confirms that if the L2R and RL components are convergent, the resulting summaries are guaranteed to be near-optimal.

Experimental Results

The authors evaluated RELIS on the DUC ’01, ’02, and ’04 multi-document summarization benchmarks, which feature clusters of documents and human-generated reference summaries.

Efficiency and Speedup

The most striking result is the efficiency gain. RELIS reduces training time by approximately two orders of magnitude compared to existing state-of-the-art models.

Training and test time comparison between RELIS and baseline models.

As shown in the timing data, while models like SRSum require over 800 seconds for training, the RELIS training phase is completed in just 2 seconds. Although test-time RL adds some latency (approx. 34 seconds), the overall pipeline remains highly efficient compared to the heavy data requirements of cross-input training.

Performance and Reward Quality

Despite its speed, RELIS does not sacrifice accuracy. It performs on par with or better than complex neural models like TCSum and SRSum, and significantly outperforms previous RL-based approaches such as DeepTD and REAPER. This success is attributed to the quality of the learned reward; the L2R-learned oracle achieved a Spearman’s correlation ($\rho$) of .601 with ground-truth ROUGE, nearly doubling the correlation of previous handcrafted rewards (e.g., REAPER’s .316).

Conclusion

RELIS demonstrates that "good rewards are all you need" for efficient RL in summarization. By shifting the heavy lifting of knowledge acquisition into a reward-learning phase, the framework enables fast and effective local optimization. While the current implementation relies on handcrafted features, the success of linear models in this context suggests that document-dependent context is a more critical factor for reward modeling than raw sequence encoding. Future work may explore automated feature learning and extending this paradigm to other sequence generation tasks.

Summary updated

BibTeX
@inproceedings{gao2019good,
  author = {Yingfang Gao and Christian M. Meyer and Mohsen Mesgar and Iryna Gurevych},
  title = {{Good Rewards Are All You Need: Reward Learning for Efficient Reinforcement Learning in Document Summarisation}},
  booktitle = {TUbilio},
  year = {2019},
}

Loading related content...

Figure