Abstract
The ReDecode framework introduces an iterative refinement process for paraphrase generation, departing from the traditional "one-shot" sequence-to-sequence approach. By employing a chain of decoders where each subsequent stage "re-reads" and improves upon the previous output, the model achieves substantial performance gains. Notably, it demonstrates a 28% absolute increase in METEOR scores on the MSCOCO dataset, highlighting the efficacy of iterative revision in neural text generation.
Introduction
Paraphrasing—the task of restating a sentence while preserving its semantic meaning—is critical for natural language processing (NLP) applications such as query expansion, conversational AI, and data augmentation. Traditional sequence generation models, including standard Seq2Seq architectures and basic Variational Autoencoders (VAEs), typically operate in a single pass. This "one-shot" approach suffers from two primary limitations: a lack of revision capabilities and the risk of error propagation, where early mistakes in the decoding process lead to syntactically or semantically flawed outputs.
To address these challenges, the ReDecode framework draws inspiration from human writing processes, where drafts are iteratively edited and refined. The authors propose a multi-stage decoding architecture that allows the model to revisit and improve its own generated text.
Methodology
The ReDecode framework is built upon a Supervised Variational Autoencoder (VAE-S) architecture, comprising three primary components:
- Sampling Encoder ($S_{\theta_1}$): This component maps the input sentence to a latent probability distribution. A latent vector $z$ is sampled from this space, capturing the "semantic gist" of the input while encouraging diversity in the generated paraphrases.
- Sentence Encoder ($E_{\theta_2}$): A two-layer Long Short-Term Memory (LSTM) network that generates a memory bank of the input sentence, which the decoders utilize via attention mechanisms.
- Iterative Decoders ($Dec_{\phi_i}$): This core innovation involves a sequence of decoders. The first decoder generates an initial "crude" paraphrase by attending to the input sentence. Subsequent decoders attend not only to the input but also to the softmax vectors (probability distributions over the vocabulary) produced by the preceding decoder.
By passing softmax vectors instead of discrete tokens between stages, the model preserves "uncertainty" from previous steps, allowing later decoders to make more informed decisions about where to adhere to the initial draft and where to pivot toward better word choices. Additionally, a MultiSample Loss is employed to minimize cosine similarity between outputs from different latent samples, ensuring expressive variety.
Experimental Setup
The framework was evaluated on two benchmark datasets:
- Quora Question Pairs: Approximately 87,000 pairs of community-asked questions labeled as duplicates.
- MSCOCO: An image captioning dataset containing multiple human-written descriptions for the same image, providing natural paraphrase pairs. The authors utilized roughly 149,000 pairs.
Both datasets were filtered to include sentences with a maximum length of 15 words.
Results and Discussion
The ReDecode framework significantly outperforms previous state-of-the-art models, such as VAE-SVG and Residual LSTMs, across multiple metrics. On the Quora dataset, it achieved a METEOR score of 41.95 and a BLEU score of 61.23. The results on MSCOCO were even more pronounced, with a METEOR score of 59.88 and a BLEU score of 87.71, representing a massive leap from previous benchmarks (30.9 METEOR / 41.3 BLEU).
The dramatic reduction in the Translation Edit Rate (TER)—dropping from 40.8 to 5.8 on MSCOCO—indicates that the generated outputs are much closer to human-written references. Qualitative analysis confirms that subsequent decoders act as effective editors, refining generic word choices into more contextually appropriate ones.
Interestingly, the study found diminishing returns with increased depth. While three decoders were optimal for the Quora dataset, two decoders performed better on MSCOCO, suggesting that the ideal number of revision passes is data-dependent.
Conclusion and Future Work
The ReDecode framework demonstrates that iterative refinement is a powerful paradigm for paraphrase generation. However, several avenues for future research remain. Currently, the number of decoding iterations is a fixed hyperparameter; a dynamic decoding approach where the model determines when to stop refining could improve efficiency. Furthermore, the computational overhead of multiple decoders and the framework's generalizability to other tasks like machine translation warrant further investigation.
Summary updated