Abstract
Unsupervised methods are promising for abstractive text summarization in that the parallel corpora is not required. However, their performance is still far from being satisfied. In this paper, we propose a new approach based on Q-learning with an edit-based summarization. The method combines two key modules to form an Editorial Agent and Language Model converter (EALM). The agent predicts edit actions (e.g., delete, keep, and replace), and then the LM converter deterministically generates a summary on the basis of the action signals. Q-learning is leveraged to train the agent to produce proper edit actions. Experimental results show that EALM delivered competitive performance compared with the previous encoder-decoder-based methods, even with truly zero paired data (i.e., no validation set). Defining the task as Q-learning enables us not only to develop a competitive method but also to make the latest techniques in reinforcement learning available for unsupervised summarization.
Introduction
Abstractive summarization typically relies on large-scale supervised datasets, which are costly and time-consuming to produce. Unsupervised alternatives often utilize the Compression-Reconstruction (CR) paradigm, where a model learns to compress a sentence and then reconstruct it. However, traditional generative encoder-decoder models in this space often suffer from exposure bias, repetitive text, and poor length control. To address these stability issues, we introduce EALM (Editorial Agent and Language Model converter), which shifts the task from pure generation to an edit-based reinforcement learning problem.
The EALM Framework
The core of EALM is the separation of decision-making and text generation. Instead of generating a sequence from scratch, the system learns to edit the existing input.
As illustrated in the comparison above, EALM consists of two primary components:
- Editorial Agent: A Deep Q-Network (DQN) that evaluates each word in the input sentence and selects an action:
Keep,Remove, orReplace. - Language Model (LM) Converter: A deterministic module that processes these actions. When a word is marked for replacement, a pre-trained Masked Language Model (BERT) predicts the most suitable replacement based on the surrounding context.
Edit-based Summarization Mechanism
The summarization process follows a structured pipeline from the original input $x$ to the final summary $\hat{x}$. This approach forces the model to stay grounded in the source text, avoiding common generative pitfalls like hallucination.
In this workflow, the agent determines the action sequence $a$. A preprocess conversion creates an intermediate state $z$. The LM converter then fills in the gaps for Replace actions to produce the compressed summary $y$. Finally, a reconstruction process attempts to recover the original sentence, providing a signal for the reconstruction reward used during training.
Training via Q-Learning
The agent is trained using a hybrid reward function that balances compression (shortness), reconstruction (retaining original meaning), and fluency (using log-likelihood from a language model).
A key innovation is confidence-based ordering. Unlike standard models that generate text left-to-right, the EALM agent can decide the order in which it predicts actions based on the highest Q-values. This allows it to handle variable sentence lengths more naturally and manage the complexity of the reinforcement learning search space.
Key Results and Insights
EALM was evaluated on standard benchmarks including Gigaword (GIGA), DUC 2003, and DUC 2004. Key findings include:
- Competitive Performance: EALM outperformed previous unsupervised methods like SEQ3 and CMatch on DUC datasets in terms of ROUGE-1 and ROUGE-L scores.
- Zero-Paired Stability: The model achieved these results with "truly zero" paired data, requiring no validation set for hyperparameter tuning.
- Keyword Retention: Qualitative analysis indicates that EALM is particularly adept at identifying and preserving keywords from the entire input sentence, avoiding the "lead bias" common in many summarizers.
Conclusion and Future Work
By framing unsupervised summarization as a Q-learning task over edit actions, EALM provides a stable and effective alternative to traditional generative models. While the model currently faces challenges with complex rephrasing and occasional grammatical artifacts, it demonstrates the potential of applying advanced reinforcement learning techniques to the field of text summarization. Future work will focus on more sophisticated reward shaping to encourage tighter summaries without sacrificing information.
Summary updated