Q-learning with Language Model for Edit-based Unsupervised Summarization

Ryosuke Kohita, Akifumi Wachi, Yang Zhao, Ryuki Tachibana

Abstract

Unsupervised methods are promising for abstractive text summarization in that the parallel corpora is not required. However, their performance is still far from being satisfied, therefore research on promising solutions is on-going. In this paper, we propose a new approach based on Q-learning with an edit-based summarization. The method combines two key modules to form an Editorial Agent and Language Model converter (EALM). The agent predicts edit actions (e.t., delete, keep, and replace), and then the LM converter deterministically generates a summary on the basis of the action signals. Q-learning is leveraged to train the agent to produce proper edit actions. Experimental results show that EALM delivered competitive performance compared with the previous encoder-decoder-based methods, even with truly zero paired data (i.e., no validation set). Defining the task as Q-learning enables us not only to develop a competitive method but also to make the latest techniques in reinforcement learning available for unsupervised summarization. We also conduct qualitative analysis, providing insights into future study on unsupervised summarizers.

1. Introduction

Abstractive summarization aims to generate concise versions of documents while preserving core meaning. Traditional supervised approaches rely on massive parallel corpora, which are often unavailable for niche domains. Unsupervised alternatives typically employ a "compression-reconstruction" cycle using encoder-decoder architectures. However, these models often suffer from exposure bias, repetitive outputs, and training instability.

To address these issues, we introduce EALM (Editorial Agent and Language Model), a framework that treats summarization as a reinforcement learning (RL) problem. Instead of generating text token-by-token, EALM utilizes an agent to edit the source text directly.

Comparison between conventional encoder-decoder approaches and the proposed EALM framework.

2. Methodology: The EALM Framework

The EALM architecture consists of two primary components: an Editorial Agent and a Language Model (LM) Converter.

2.1 The Editorial Agent

The agent is trained via Deep Q-Learning (DQN) to predict one of three discrete actions for each word in the input: Keep, Remove, or Replace. Unlike standard sequence models that operate left-to-right, the EALM agent uses Q-values to determine an optimal editing order, prioritizing words it is most "confident" about first. This non-linear processing allows the model to refine its state based on high-confidence edits before tackling more ambiguous words.

2.2 The LM Converter and BERT Integration

The LM Converter translates the agent's action signals into a final summary. For Replace actions, the framework leverages a pre-trained Masked Language Model (BERT). By treating BERT as a "converter" that fills in [MASK] tokens, the system can introduce novel synonyms and more concise phrasing without the need for a massive generative vocabulary during RL training.

Example of the edit-based summarization process using the Editorial Agent and LM Converter.

2.3 Reward Function

The agent is optimized using a multi-faceted reward function that balances:

  • Compression Rate: Encouraging shorter outputs.
  • Reconstruction Rate: Ensuring the original sentence can be recovered.
  • Semantic Similarity: Maintaining the original meaning.
  • Fluency: Using log-likelihood from the LM to ensure natural-sounding language.

3. Experiments and Results

We evaluated EALM on the Gigaword news corpus and the DUC 2003/2004 benchmarks. Notably, EALM can be trained with "truly zero" paired data, as it does not require a labeled validation set for model selection.

3.1 Quantitative Performance

EALM demonstrated superior generalization compared to baselines like SEQ3 and CMatch. On DUC 2003, EALM achieved a ROUGE-1 score of 21.69, significantly outperforming CMatch's 17.50. The use of a discrete action space contributed to greater training stability compared to traditional generative models.

3.2 Qualitative Analysis

Analysis revealed that the Replace action successfully introduced vocabulary innovations, though less frequently than human editors. However, some summaries exhibited minor grammatical issues, such as the deletion of necessary prepositions. Additionally, the agent occasionally produced longer summaries due to a conservative approach toward the reconstruction reward.

4. Conclusion

EALM offers a robust alternative to generative unsupervised summarization by framing the task as an edit-based RL problem. By decoupling the editorial logic from the linguistic generation (via BERT), the model achieves competitive results and improved stability. Future work will explore more advanced RL techniques and refined fluency constraints to further enhance grammatical accuracy.

Summary updated

Abstract

Unsupervised methods are promising for abstractive textsummarization in that the parallel corpora is not required. However, their performance is still far from being satisfied, therefore research on promising solutions is on-going. In this paper, we propose a new approach based on Q-learning with an edit-based summarization. The method combines two key modules to form an Editorial Agent and Language Model converter (EALM). The agent predicts edit actions (e.t., delete, keep, and replace), and then the LM converter deterministically generates a summary on the basis of the action signals. Q-learning is leveraged to train the agent to produce proper edit actions. Experimental results show that EALM delivered competitive performance compared with the previous encoder-decoder-based methods, even with truly zero paired data (i.e., no validation set). Defining the task as Q-learning enables us not only to develop a competitive method but also to make the latest techniques in reinforcement learning available for unsupervised summarization. We also conduct qualitative analysis, providing insights into future study on unsupervised summarizers.
BibTeX
@inproceedings{kohita2020q,
  author = {Ryosuke Kohita and Akifumi Wachi and Yang Zhao and Ryuki Tachibana},
  title = {{Q-learning with Language Model for Edit-based Unsupervised Summarization}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=GMo6z3lm35E},
}
Figure