Abstract
Paraphrase generation (PG) is often limited by a "data drought," where the lack of large-scale, high-quality corpora hinders the training of deep neural networks. This paper demonstrates that sequential transfer learning can overcome this challenge. By pre-training a Sequence-to-Sequence (SEQ2SEQ) model on large textual entailment (SNLI) and phrasal paraphrase (PPDB) datasets before fine-tuning on the target Microsoft Research Paraphrase (MSRP) corpus, performance is significantly boosted. The best pipeline, SNLI -> PPDB -> MSRP, achieves a BLEU score of 15.76, a dramatic improvement over the 0.09 baseline.
Introduction
In Natural Language Processing (NLP), paraphrasing—restating a sentence while preserving its meaning—is a foundational task useful for question expansion, sentence simplification, and more. While Neural Machine Translation (NMT) has flourished due to massive bilingual datasets, PG has lagged behind. Collecting pairs of complex sentences with identical meanings is difficult and expensive. The "gold standard" MSRP corpus contains fewer than 4,000 training pairs, which is far too few to train a modern SEQ2SEQ model from scratch without massive overfitting.
Methods
The authors treat paraphrasing as a monolingual translation task using a standard NMT architecture: a SEQ2SEQ model with an attention mechanism. The model employs a 2-layer LSTM with 2000 hidden units and word embeddings of size 1000.
To address the data scarcity, the authors propose sequential transfer learning. They hypothesize that Entailment Generation (EG)—generating a sentence logically inferred from another—is a "close cousin" to paraphrasing. The training pipeline follows a specific multi-stage "hop" pattern:
- Stage 1: Train on the large SNLI entailment dataset.
- Stage 2: Fine-tune on the medium-sized PPDB phrasal paraphrase dataset.
- Stage 3: Final fine-tune on the target MSRP complex sentence dataset.
Datasets
The study utilizes three primary datasets to bridge the data gap:
- SNLI (Stanford Natural Language Inference): ~183,000 entailment pairs. While entailment is directional, it provides a wealth of lexical overlap data.
- PPDB (Paraphrase Database): The "XS" subset containing ~457,000 high-quality phrasal paraphrase pairs.
- MSRP (Microsoft Research Paraphrase): The target evaluation set, featuring ~3,800 training pairs of complex sentences.
Data augmentation was also performed by "flipping" the pairs, which is logically sound for paraphrases and helps the model learn bidirectional relationships.
Results
The quantitative improvements were dramatic compared to the baseline. As shown in Table 1, a model trained only on MSRP achieved a dismal BLEU score of 0.09. By using the full transfer learning pipeline (SNLI -> PPDB -> MSRP), the score jumped to 15.76, and the perplexity dropped to 3.08, indicating much more confident and accurate word predictions.
Qualitatively, the model proved capable of restating parts of input sentences accurately. However, as seen in Table 2, it occasionally struggled with "finer details" or proper nouns. For example, the model might drop specific names or fail to capture the full complexity of the target sentence.
Discussion and Limitations
Despite the gains, several hurdles remain:
- Lexical Variety: The models often play it "safe," failing to show significant lexical variation or use diverse synonyms.
- The "Proper Noun" Problem: Without a specific "copying mechanism," the model struggles to handle rare words or proper names that were infrequent in the training data.
- The Truncation Effect: Models trained on SNLI tend to generate shorter sentences because SNLI "hypotheses" are naturally shorter than their "premises," leading to an inadvertent summarization bias.
- Meaning Loss: In complex sentences, the model sometimes drops crucial words, which fundamentally alters the meaning.
Conclusion
Sequential transfer learning from entailment and phrasal paraphrase tasks is an effective strategy for training paraphrase generators on small datasets. The SNLI -> PPDB -> MSRP pipeline leverages the logic of entailment and the semantic equivalence of phrases to improve full-sentence paraphrasing. Future work will focus on incorporating copying mechanisms and exploring ways to encourage more natural lexical diversity.
Summary updated