Neural Paraphrase Generation using Transfer Learning

Florin Brad, Traian Rebedea

Abstract

Paraphrase generation (PG) is often limited by a "data drought," where the lack of large-scale, high-quality corpora hinders the training of deep neural networks. This paper demonstrates that sequential transfer learning can overcome this challenge. By pre-training a Sequence-to-Sequence (SEQ2SEQ) model on large textual entailment (SNLI) and phrasal paraphrase (PPDB) datasets before fine-tuning on the target Microsoft Research Paraphrase (MSRP) corpus, performance is significantly boosted. The best pipeline, SNLI -> PPDB -> MSRP, achieves a BLEU score of 15.76, a dramatic improvement over the 0.09 baseline.

Introduction

In Natural Language Processing (NLP), paraphrasing—restating a sentence while preserving its meaning—is a foundational task useful for question expansion, sentence simplification, and more. While Neural Machine Translation (NMT) has flourished due to massive bilingual datasets, PG has lagged behind. Collecting pairs of complex sentences with identical meanings is difficult and expensive. The "gold standard" MSRP corpus contains fewer than 4,000 training pairs, which is far too few to train a modern SEQ2SEQ model from scratch without massive overfitting.

Methods

The authors treat paraphrasing as a monolingual translation task using a standard NMT architecture: a SEQ2SEQ model with an attention mechanism. The model employs a 2-layer LSTM with 2000 hidden units and word embeddings of size 1000.

To address the data scarcity, the authors propose sequential transfer learning. They hypothesize that Entailment Generation (EG)—generating a sentence logically inferred from another—is a "close cousin" to paraphrasing. The training pipeline follows a specific multi-stage "hop" pattern:

  1. Stage 1: Train on the large SNLI entailment dataset.
  2. Stage 2: Fine-tune on the medium-sized PPDB phrasal paraphrase dataset.
  3. Stage 3: Final fine-tune on the target MSRP complex sentence dataset.

Datasets

The study utilizes three primary datasets to bridge the data gap:

  • SNLI (Stanford Natural Language Inference): ~183,000 entailment pairs. While entailment is directional, it provides a wealth of lexical overlap data.
  • PPDB (Paraphrase Database): The "XS" subset containing ~457,000 high-quality phrasal paraphrase pairs.
  • MSRP (Microsoft Research Paraphrase): The target evaluation set, featuring ~3,800 training pairs of complex sentences.

Data augmentation was also performed by "flipping" the pairs, which is logically sound for paraphrases and helps the model learn bidirectional relationships.

Results

The quantitative improvements were dramatic compared to the baseline. As shown in Table 1, a model trained only on MSRP achieved a dismal BLEU score of 0.09. By using the full transfer learning pipeline (SNLI -> PPDB -> MSRP), the score jumped to 15.76, and the perplexity dropped to 3.08, indicating much more confident and accurate word predictions.

Table 1: Quantitative results showing BLEU scores and perplexity across different transfer learning configurations.

Qualitatively, the model proved capable of restating parts of input sentences accurately. However, as seen in Table 2, it occasionally struggled with "finer details" or proper nouns. For example, the model might drop specific names or fail to capture the full complexity of the target sentence.

Table 2: Qualitative examples of source, decoded, and target sentences.

Discussion and Limitations

Despite the gains, several hurdles remain:

  • Lexical Variety: The models often play it "safe," failing to show significant lexical variation or use diverse synonyms.
  • The "Proper Noun" Problem: Without a specific "copying mechanism," the model struggles to handle rare words or proper names that were infrequent in the training data.
  • The Truncation Effect: Models trained on SNLI tend to generate shorter sentences because SNLI "hypotheses" are naturally shorter than their "premises," leading to an inadvertent summarization bias.
  • Meaning Loss: In complex sentences, the model sometimes drops crucial words, which fundamentally alters the meaning.

Conclusion

Sequential transfer learning from entailment and phrasal paraphrase tasks is an effective strategy for training paraphrase generators on small datasets. The SNLI -> PPDB -> MSRP pipeline leverages the logic of entailment and the semantic equivalence of phrases to improve full-sentence paraphrasing. Future work will focus on incorporating copying mechanisms and exploring ways to encourage more natural lexical diversity.

Summary updated

BibTeX
@inproceedings{brad2017neural,
  author = {Florin Brad and Traian Rebedea},
  title = {{Neural Paraphrase Generation using Transfer Learning}},
  booktitle = {INLG},
  year = {2017},
  doi = {10.18653/v1/w17-3542},
}
Figure