Introduction
Paraphrase generation—the task of rewriting a sentence while maintaining its semantic essence—is a fundamental challenge in Natural Language Processing (NLP). Its applications range from enhancing Question Answering (QA) systems to data augmentation for machine learning. Traditional approaches, primarily based on sequence-to-sequence (Seq2Seq) models, often struggle with generating diverse outputs, frequently producing minor surface-level variations rather than meaningful structural changes.
The Challenge of Diversity and Conditioning
The authors identify two primary limitations in current research:
- Lack of Diversity: Standard Seq2Seq models rely on beam search, which tends to yield highly similar candidates.
- Unconditional Generation: While Variational Autoencoders (VAEs) are powerful generative tools, standard versions generate text from a latent space without direct conditioning on a specific input, making them unsuitable for targeted paraphrasing.
Methodology: The VAE-SVG Framework
To address these issues, the paper introduces a VAE-LSTM architecture, termed VAE-SVG (Sentence Variant Generation). The core innovation lies in a dual-conditioning mechanism that operates on both the encoding and decoding stages.
Conditioning Mechanism
- Encoding Side: The model employs two LSTMs. The first encodes the original sentence ($s^{(o)}$) into a vector $x^{(o)}$. The second encodes the target paraphrase ($s^{(p)}$) while simultaneously conditioning on $x^{(o)}$. This ensures the latent space $z$ captures the specific relationship and variations between the pair.
- Decoding Side: During reconstruction, the decoder is fed both the latent code $z$ and the original sentence representation $x^{(o)}$. This allows $x^{(o)}$ to preserve the core meaning while $z$ introduces generative diversity.
The authors also explored a parameter-efficient variant, VAE-SVG-eq, which shares the LSTM for the original sentence across both sides of the model.
Experimental Setup and Results
The framework was evaluated on two benchmark datasets: MSCOCO (using image captions as paraphrases) and Quora Question Pairs (a large-scale dataset of duplicate questions).
Quantitative Performance
The VAE-SVG model demonstrated significant improvements over existing state-of-the-art methods:
- MSCOCO: Achieved a BLEU score of 41.3, surpassing the previous best of 37.0. It also showed improved Translation Error Rate (TER) scores.
- Quora: Reached a BLEU score of 38.3 using beam search, a massive leap from the 11.4 score of unsupervised VAE baselines.
Qualitative and Human Evaluation
Human evaluators rated the outputs on relevance and readability. For the Quora dataset, the model achieved scores of 3.57 for Relevance and 4.08 for Readability (on a 1-5 scale). Qualitatively, the model showed an ability to infer context; for instance, expanding "A man with luggage" into "A young man standing in front of an airport," demonstrating a deep semantic understanding beyond simple word replacement.
Conclusion
By integrating VAEs with a robust conditioning mechanism, the VAE-SVG framework provides a principled way to generate diverse and semantically accurate paraphrases. The ability to sample from a latent Gaussian distribution offers a more generative alternative to traditional search-based methods. Future work may involve extending this modular framework to controllable text generation tasks, such as adjusting the formality or length of generated paraphrases.
Summary updated