Reinforcement Replaces Supervision: Query focused Summarization using Deep Reinforcement Learning

Swaroop Nath, Pushpak Bhattacharyya, Harshad KhadilkarEMNLP 2023 · DOI

Introduction

Query-focused Summarization (QfS) is a critical task in natural language processing that requires a system to generate a concise summary of a document in direct response to a specific user query. Traditionally, these models are trained using Supervised Learning (SL) with a technique known as "Teacher Forcing." While effective for standard language modeling, this paper argues that SL is fundamentally limited for QfS tasks. The authors propose a Reinforcement Learning (RL) framework that significantly outperforms state-of-the-art supervised models by optimizing for semantic quality rather than simple token prediction.

The Limitations of Supervised Learning

The researchers identify two primary failures in the standard SL approach:

  1. Exposure Bias: During training, SL models are always provided with the correct "gold standard" token at each step. However, during real-world inference, the model must rely on its own previously generated (and potentially erroneous) tokens. A single small mistake can lead to a cascade of errors because the model was never trained to recover from its own deviations.
  2. Lexical Rigidity: SL rewards the model for predicting the exact words found in a reference summary. This ignores the fact that a high-quality summary can be written in many different ways. By focusing on word-level mimicry, SL models fail to capture the broader semantic intent of the query.

Methodology and the SfPEG Reward

To address these issues, the authors developed an RL framework using Policy Gradient networks. A major technical challenge in this domain is the conflict between the sequential nature of RL and the parallel processing architecture of Transformers. The authors resolved this using Scheduled Sampling and a "two-pass" decoder strategy, allowing the model to learn from its own generated sequences without the prohibitive computational costs typically associated with RL in Transformers.

Central to their approach is a novel reward function: Similarity From Passage EmbeddinG (SfPEG). This reward is based on the "Cluster Hypothesis," which posits that passages satisfying similar information needs should be close to one another in a high-dimensional vector space.

Table 2: Definitions and formulas for the various reward functions evaluated, including the novel SfPEG.

By training a separate passage embedder on a massive new dataset (RPEDT, containing ~8 million Reddit samples), the authors created a reward signal that prioritizes semantic similarity over simple word overlap (like ROUGE or BLEU).

Key Insights: RL Models Actually "Read"

One of the paper's most compelling findings came from a "Random Document" experiment. When the correct document was replaced with a random one, SL models continued to generate answers based on the query alone—essentially hallucinating. In contrast, the performance of the RL models plummeted. This demonstrates that the RL objective forces the model to actually utilize and ground its summary in the provided document, whereas SL models often simply memorize query-answer patterns.

Results and Evaluation

The RL-based model, specifically BART R-SfPEG, demonstrated a massive 10.62 point improvement in ROUGE-L over the BART SL baseline on the ELI5 benchmark. Furthermore, the model showed remarkable zero-shot capabilities, matching or exceeding the performance of fine-tuned models on the DebatePedia dataset.

Human evaluation confirmed these automatic metrics. Annotators rated the RL-trained models significantly higher in both fluency and correctness. Notably, the BART R-SfPEG model achieved a 19% improvement in Correctness over the supervised baseline, proving that the model is not just more fluent, but more factually grounded in the source text.

Table 6: Human evaluation results comparing model outputs on Fluency and Correctness.

Conclusion and Future Work

This research proves that Reinforcement Learning is a superior paradigm for Query-focused Summarization. By moving away from token-level mimicry and toward semantic reward optimization, the authors achieved a new state-of-the-art. While the training process remains computationally intensive—requiring 21 days on 8x A100 GPUs—the results suggest that the future of summarization lies in models that are rewarded for the quality of their output rather than the accuracy of their next-word predictions.

Summary updated

BibTeX
@inproceedings{nath2023reinforcement,
  author = {Swaroop Nath and Pushpak Bhattacharyya and Harshad Khadilkar},
  title = {{Reinforcement Replaces Supervision: Query focused Summarization using Deep Reinforcement Learning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.977},
}
Figure