Contrastive Representation Learning for Exemplar-Guided Paraphrase Generation

Haoran Yang, Wai Lam, Piji Li

Abstract

Exemplar-Guided Paraphrase Generation (EGPG) aims to rewrite a source sentence to match the syntactic style of an exemplar while preserving original meaning. A major hurdle in this task is the effective disentanglement of content and style. This paper introduces a novel framework utilizing contrastive learning to explicitly supervise the separation of these features. By incorporating Content Contrastive Loss (CCL) and Style Contrastive Loss (SCL), the model achieves state-of-the-art performance on benchmark datasets, significantly reducing content hallucination and improving stylistic fidelity.

Introduction

Paraphrasing with specific stylistic constraints is a complex task for AI. In EGPG, the model receives a source sentence ($X$) and an exemplar ($Z$), producing a target ($Y$) that mirrors $X$'s meaning and $Z$'s syntax. Previous approaches often suffered from "topic drift," where the model inadvertently incorporates content from the exemplar into the output. This failure stems from the inability of standard encoders to perfectly disentangle semantic content from syntactic structure.

Methodology: Contrastive Disentanglement

The proposed architecture addresses disentanglement through a dual-encoder system enhanced by contrastive objectives.

The proposed architecture featuring dual encoders ($E_s$ for style, $E_c$ for content) and the integration of style and content contrastive losses.

The framework consists of:

  1. Dual Encoders: A BERT-based encoder ($E_s$) extracts stylistic features, while a GRU-based encoder ($E_c$) captures semantic content.
  2. Contrastive Losses:
    • Content Contrastive Loss (CCL): Ensures the embeddings of the source ($X$) and target ($Y$) are close, while pushing away unrelated sentences.
  3. Style Contrastive Loss (SCL): Aligns the style embeddings of the exemplar ($Z$) and the target ($Y$), distinguishing them from different syntactic patterns.
  4. Joint Optimization: The model is trained by minimizing a weighted sum of the standard generation loss (NLL) and the two contrastive losses. This multi-task setup provides stronger supervision signals, mitigating gradient vanishing and forcing the encoders to learn more robust, task-specific features.

Experimental Results

The model was evaluated on the QQP-Pos and ParaNMT datasets. To create training pairs, the authors utilized POS-tag similarity to identify appropriate exemplars.

Performance comparison of the proposed model against baselines (SCPN, SGCP, CGEN) and ablation variants across multiple metrics.

As shown in the results table, the contrastive approach yields substantial improvements:

  • State-of-the-Art Performance: On QQP-Pos, the model achieved a BLEU score of 45.8, a nearly 25% relative improvement over the previous best baseline (SGCP at 36.7).
  • Ablation Insights: Removing either CCL or SCL leads to a measurable decline in performance, confirming that both semantic and stylistic supervision are critical.
  • Content Fidelity: The authors introduced a Content Matching Accuracy (CMA) metric, demonstrating that their method significantly reduces "topic confusion"—a common issue where models confuse similar entities due to contextual similarities.

Conclusion

By leveraging contrastive learning as a regularizer, this work provides a powerful solution for feature disentanglement in text generation. The resulting model produces paraphrases that are both syntactically accurate to the exemplar and semantically faithful to the source. Future research may extend this framework to more abstract stylistic dimensions, such as sentiment or formality, and explore its application in broader controlled generation tasks.

Summary updated

Abstract

Exemplar-Guided Paraphrase Generation (EGPG) aims to generate a target sentence which conforms to the style of the given exemplar while encapsulating the content information of the source sentence. In this paper, we propose a new method with the goal of learning a better representation of the style andthe content. This method is mainly motivated by the recent success of contrastive learning which has demonstrated its power in unsupervised feature extraction tasks. The idea is to design two contrastive losses with respect to the content and the style by considering two problem characteristics during training. One characteristic is that the target sentence shares the same content with the source sentence, and the second characteristic is that the target sentence shares the same style with the exemplar. These two contrastive losses are incorporated into the general encoder-decoder paradigm. Experiments on two datasets, namely QQP-Pos and ParaNMT, demonstrate the effectiveness of our proposed constrastive losses.
BibTeX
@inproceedings{yang2021contrastive,
  author = {Haoran Yang and Wai Lam and Piji Li},
  title = {{Contrastive Representation Learning for Exemplar-Guided Paraphrase Generation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.findings-emnlp.409},
}
Figure