An End-to-End Generative Architecture for Paraphrase Generation

Qian Yang, Zhouyuan Huo, Dinghan Shen, Yong Cheng, Wenlin Wang, Guoyin Wang, Lawrence CarinEMNLP 2019 · OpenReview

Abstract

Qian Yang, Zhouyuan Huo, Dinghan Shen, Yong Cheng, Wenlin Wang, Guoyin Wang, Lawrence Carin. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).

2019.

Introduction

Paraphrase generation—the task of rewriting a sentence while preserving its original meaning—is a fundamental challenge in Natural Language Processing (NLP). It serves as a critical component for applications such as dialogue systems, question answering, and data augmentation. Despite its importance, existing generative models face three primary obstacles:

  1. Exposure Bias: Traditional Recurrent Neural Networks (RNNs) trained with "teacher forcing" suffer from a discrepancy between training and inference, where errors accumulate during autonomous generation.
  2. Latent Space "Dead Zones": Conditional Variational Autoencoders (CVAEs) often learn latent spaces where certain regions decode into nonsensical text.
  3. Heavy Dependencies: Many state-of-the-art models rely on external linguistic tools like syntactic parsers or semantic dictionaries, which limit their scalability and generalizability.

To address these issues, we introduce the Generative Adversarial Paraphrase (GAP) model, an end-to-end architecture that integrates CVAEs with Generative Adversarial Networks (GANs).

Methods

The GAP model treats a CVAE as the generator within a GAN framework, utilizing a novel "two-path" loss and adversarial training to produce realistic and diverse paraphrases without external linguistic dependencies.

The GAP architecture featuring a two-path encoder and adversarial training components.

The architecture consists of three primary modules:

  • The Encoder ($E$): Employs a unique two-path structure. During training, one path processes both source and target sentences to learn the latent distribution, while the second path processes only the source sentence. This design ensures the model remains robust during inference when the target is unknown.
  • The Generator ($G$): Composed of two LSTMs. The first encodes the source sentence, and the second decodes a latent variable $z$ (sampled via the reparameterization trick) into the final paraphrase.
  • The Discriminator ($D$): An LSTM-based classifier that distinguishes between ground-truth human paraphrases and those produced by the generator.

Training is stabilized through a warm-up phase and uses Policy Gradient (REINFORCE) to backpropagate signals from the discriminator to the generator. This adversarial setup acts as a global critic, forcing the model to generate sentences that are naturally coherent rather than just word-perfect, effectively mitigating exposure bias.

Key Results

The GAP model was evaluated on the MSCOCO and Quora Question Pairs datasets, outperforming several strong baselines including standard Seq2Seq models and Transformers.

Quantitative Performance

On the MSCOCO dataset, GAP achieved a BLEU score of 45.60, significantly higher than the Transformer-PB baseline (44.00) and previous VAE models (41.70). The model also showed strong scalability across Quora subsets of 50k, 100k, and 150k pairs.

Qualitative and Human Evaluation

Human evaluations (conducted via Amazon Mechanical Turk) rated GAP higher than previous VAE models across three metrics: Plausibility, Equivalence, and Diversity. Notably, the model demonstrated an ability to select more appropriate synonyms than the ground truth in some instances, such as correctly using "resign" as a synonym for "quitting."

Qualitative comparison of generated paraphrases against ground truth and standard VAE models.

Conclusion

The GAP model represents a significant step forward in end-to-end paraphrase generation. By combining the global representation capabilities of CVAEs with the adversarial rigor of GANs, it effectively mitigates exposure bias and produces high-quality text. Future work will explore adapting this adversarial approach to Transformer-based backbones and extending it to other conditional generation tasks like style transfer or dialogue response generation.

Summary updated

BibTeX
@inproceedings{yang2019end,
  author = {Qian Yang and Zhouyuan Huo and Dinghan Shen and Yong Cheng and Wenlin Wang and Guoyin Wang and Lawrence Carin},
  title = {{An End-to-End Generative Architecture for Paraphrase Generation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=Mgd5L-lXYFv},
}
Figure