Abstract
We introduce a novel schema for sequence to sequence learning with a Deep Q-Network (DQN), which decodes the output sequence iteratively. The aim here is to enable the decoder to first tackle easier portions of the sequences, and then turn to cope with difficult parts. Specifically, in each iteration, an encoder-decoder Long Short-Term Memory (LSTM) network is employed to, from the input sequence, automatically create features to represent the internal states of and formulate a list of potential actions for the DQN. Take rephrasing a natural sentence as an example. This list can contain ranked potential words. Next, the DQN learns to make decision on which action (e.g., word) will be selected from the list to modify the current decoded sequence. The newly modified output sequence is subsequently used as the input to the DQN for the next decoding iteration. In each iteration, we also bias the reinforcement learning's attention to explore sequence portions which are previously difficult to be decoded. For evaluation, the proposed strategy was trained to decode ten thousands natural sentences. Our experiments indicate that, when compared to a left-to-right greedy beam search LSTM decoder, the proposed method performed competitively well when decoding sentences from the training set, but significantly outperformed the baseline when decoding unseen sentences, in terms of BLEU score obtained.
Introduction
Traditional sequence-to-sequence (seq2seq) models, prevalent in machine translation and summarization, typically generate text in a single linear pass, usually from left to right. This "one-pass" approach suffers from two primary limitations: error propagation, where an early mistake can derail the entire sequence, and a rigid structure that forces the model to resolve complex grammatical dependencies before establishing the broader context. This paper proposes a novel iterative decoding approach using Deep Q-Networks (DQN) to mimic human-like editing, allowing the model to refine sentences over multiple passes by tackling easier segments first.
Methodology
The proposed framework combines supervised learning with Reinforcement Learning (RL) through an iterative refinement process.
- State Generation (StateGF): A pre-trained LSTM encoder-decoder encodes the input and generates an initial sequence along with candidate actions (potential word replacements) for each position.
- DQN Agent: The agent "edits" the sequence iteratively. It observes the current state—comprising the original input and the current decoded version—and selects an action to improve the sequence.
- Bidirectional Context: To provide a global view, a Bidirectional LSTM represents internal states, allowing the DQN to consider both preceding and succeeding tokens when making modifications.
- Reward Signal: The agent receives a reward based on the change in the Smoothed BLEU score. Improvements yield a +1 reward, while degradations result in -1.
- Training Strategy: To ensure stability, the LSTMs are pre-trained and then frozen (or updated with a minimal learning rate) while the DQN learns the editing policy.
Experimental Results
The model was evaluated on a sentence regeneration task using 12,000 sentences from the Billion Word Corpus. The DQN approach demonstrated a remarkable ability to generalize. While performance on seen data was slightly improved, the DQN model more than doubled the BLEU score on unseen test data compared to the greedy LSTM baseline.
| System | Seen Test Set (BLEU) | Unseen Test Set (BLEU) |
|---|---|---|
| Baseline (Greedy LSTM) | 0.425 | 0.107 |
| Proposed (DQN) | 0.494 | 0.228 |
This significant gain in generalization is attributed to the DQN's exploration strategy. By encountering and correcting various "wrong" words during training, the model becomes robust to noise and novel structures.
Discussion and Conclusion
The iterative DQN-based decoding schema offers a powerful alternative to traditional greedy search, particularly for improving generalization on unseen data. However, the approach introduces computational overhead due to its multi-pass nature and is currently constrained by a limited action space. Future work will explore expanding the vocabulary of actions and applying the framework to more complex tasks like machine translation.
Summary updated