Contrastive Representation Learning for Exemplar-Guided Paraphrase Generation

Haoran Yang, Wai Lam, Piji Li

Abstract

Exemplar-Guided Paraphrase Generation (EGPG) is the task of generating a paraphrase that retains the semantic meaning of a source sentence while adopting the syntactic structure of a provided exemplar. The primary challenge lies in disentangling content from style within latent representations. We propose a novel framework that integrates contrastive learning into an encoder-decoder architecture to enforce this separation. By applying specific content and style contrastive losses, our model forces the encoders to specialize in their respective domains. Our approach demonstrates significant improvements over state-of-the-art baselines on the QQP-Pos and ParaNMT benchmarks, producing paraphrases that are more semantically accurate and stylistically consistent.

Introduction

Paraphrasing with stylistic control requires a model to merge the "what" (content) of a source sentence with the "how" (style) of an exemplar. Traditional neural models often suffer from "content drift" or "style mismatch" because they fail to properly isolate these two components. When style and content are entangled in the latent space, the decoder may inadvertently alter the meaning or ignore the target structure. This paper addresses this bottleneck by leveraging contrastive representation learning to achieve robust disentanglement.

Methodology

The proposed framework utilizes a dual-encoder setup to process the source and exemplar sentences independently.

Architecture

As illustrated in Figure 1, the model consists of a content encoder ($E_c$), a style encoder ($E_s$), and a shared decoder ($D$). The content encoder is tasked with extracting semantic features, while the style encoder captures syntactic patterns.

The proposed EGPG framework featuring dual encoders and contrastive loss objectives.

Contrastive Objectives

To ensure the encoders specialize, we introduce two auxiliary losses:

  1. Content Contrastive Loss (CCL): This objective ensures that the content embeddings of the source sentence ($X$) and its target paraphrase ($Y$) are similar, as they share the same meaning. It penalizes the model if these embeddings diverge or if they become too similar to unrelated sentences in the training batch.
  2. Style Contrastive Loss (SCL): Similarly, the SCL aligns the style embeddings of the target sentence ($Y$) with the exemplar ($Z$). This forces the model to replicate the syntactic template of the exemplar in the generated output.

The final training objective is a joint optimization of the standard generation loss (negative log-likelihood) and these two contrastive terms. This multi-task approach also helps mitigate gradient vanishing by providing more direct supervision to the encoders.

Experimental Results

We evaluated our model on two benchmark datasets: QQP-Pos (derived from Quora Question Pairs) and ParaNMT (generated via back-translation).

Quantitative Performance

Our model significantly outperformed existing baselines such as SCPN, SGCP, and CGEN across all automatic metrics. As shown in Figure 6, the improvements are particularly pronounced on the QQP-Pos dataset, where our model achieved a BLEU score of 45.8, compared to the previous best of 36.7.

Comprehensive performance comparison of our model against state-of-the-art baselines on QQP-Pos and ParaNMT.

Ablation Studies and Human Evaluation

Ablation studies confirmed that both CCL and SCL contribute to the model's success. Removing either component led to a drop in performance, particularly in Content Matching Accuracy (CMA). Furthermore, human evaluations rated our model's outputs higher for both fluency and stylistic adherence, indicating that the quantitative gains reflect genuine improvements in text quality.

Conclusion

By incorporating contrastive learning, we have developed a more effective method for disentangling content and style in exemplar-guided paraphrase generation. Our model not only achieves higher accuracy on standard benchmarks but also demonstrates better structural adherence to stylistic templates. Future work will focus on improving data quality for diverse training and extending this contrastive framework to other controllable generation tasks like sentiment-based style transfer.

Summary updated

BibTeX
@inproceedings{yang2021contrastive,
  author = {Haoran Yang and Wai Lam and Piji Li},
  title = {{Contrastive Representation Learning for Exemplar-Guided Paraphrase Generation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.findings-emnlp.409},
}
Figure