MCPG: A Flexible Multi-Level Controllable Framework for Unsupervised Paraphrase Generation

Yi Chen, Haiyun Jiang, Lemao Liu, Rui Wang, Shuming Shi, Ruifeng XuEMNLP 2022 · OpenReview

Abstract

Unsupervised paraphrase generation often struggles with maintaining factual accuracy and providing diverse outputs. We present MCPG (Multi-level Controllable Paraphrase Generator), a framework that decomposes the task into global semantics, local lexicons, and universal styles. By leveraging inference-time dropout for diversity and a training-free domain adaptation technique, MCPG significantly outperforms existing unsupervised methods and rivals supervised models on standard benchmarks.

Introduction

Paraphrase generation is a fundamental task in Natural Language Processing (NLP), involving the rewriting of a sentence to convey the same meaning using different wording. This capability is critical for applications such as data augmentation, question answering, and text simplification. Traditionally, the most effective models have relied on supervised learning using large-scale parallel corpora. However, such datasets are expensive to curate and often domain-specific.

Unsupervised paraphrase generation offers a more scalable alternative but frequently suffers from a lack of control. Existing methods often fail to preserve factual accuracy or produce repetitive, low-diversity outputs. To address these challenges, we present the Multi-level Controllable Paraphrase Generator (MCPG), a framework that mimics the human paraphrasing process by capturing the big picture, protecting key facts, and optionally adopting a specific style.

The MCPG Framework

MCPG is built upon the T5 (Text-to-Text Transfer Transformer) backbone, leveraging its pre-trained capacity for text-filling. The framework operates across three distinct levels of control:

1. Global Semantics and Inference-Time Dropout

The model captures the "big picture" meaning of an input sentence using SimCSE, a BERT-based sentence encoder. A key innovation of MCPG is the use of dropout during the inference phase. While dropout is typically a training-only regularization technique, MCPG keeps it enabled to "jitter" the semantic embeddings. This allows the generator to produce multiple, diverse paraphrases from a single input by exploring different regions of the semantic space.

2. Local Lexicons and Factual Anchors

To ensure that specific facts—such as names, dates, and organizations—are preserved, MCPG incorporates a Keyword Selector. By utilizing Named Entity Recognition (NER), the model identifies critical factual entities and treats them as hard constraints. These keywords are passed to the generator to ensure they appear in the final output, preventing the "hallucination" or omission of key details common in unsupervised models.

3. Universal Styles and Domain Adaptation

For scenarios requiring specific stylistic shifts (e.g., from a formal question to a more conversational tone), we introduce MCPG-adapt. This version utilizes a "transfer vector" approach. By retrieving a small set of exemplar pairs from a target domain, the model calculates the average difference in their semantic embeddings. This vector is then added to the input's embedding, effectively "nudging" the output toward the desired style without requiring additional training or fine-tuning.

The MCPG framework architecture, illustrating both the basic and adaptive versions.

Key Insights: Dropout as a Diversity Knob

One of the most significant findings in this work is the efficacy of dropout probability ($dp$) as a control mechanism for output diversity. By adjusting $dp$ during inference, users can tune the balance between semantic fidelity and lexical variety.

As shown in the experimental results, increasing the dropout probability leads to a decrease in Self-BLEU (indicating higher diversity) while maintaining competitive BLEU and iBLEU scores. This suggests that the model can generate fresh expressions without losing the core meaning of the original sentence.

The impact of dropout probability on diversity (SelfBleu) and quality (Bleu, iBleu) metrics.

Experimental Results

The MCPG framework was evaluated on three major benchmarks: Quora Question Pairs (QQP), MSCOCO, and Twitter.

  • Unsupervised Superiority: On the Quora dataset, MCPG-basic achieved a BLEU score of 24.84, outperforming the previous unsupervised state-of-the-art, ConRPG (18.31).
  • Competitive with Supervised Models: MCPG-adapt reached a BLEU score of 37.22 on Quora, surpassing several strong supervised baselines such as IANet+S (27.09). This is particularly noteworthy as MCPG achieves these results without any parallel training data.
  • Human Evaluation: Blind human assessments confirmed that MCPG produces more fluent and diverse paraphrases compared to existing unsupervised methods while maintaining high semantic similarity.

Conclusion

MCPG provides a flexible and powerful framework for unsupervised paraphrase generation. By separating control into semantic, lexical, and stylistic levels, it achieves a balance of diversity and accuracy that rivals supervised approaches. The introduction of inference-time dropout and training-free domain adaptation via transfer vectors offers a significant step forward in making paraphrase generation more accessible and controllable for real-world applications. Future work will explore more granular syntactic controls and the application of this framework to general text polishing tasks.

Summary updated

BibTeX
@inproceedings{chen2022mcpg,
  author = {Yi Chen and Haiyun Jiang and Lemao Liu and Rui Wang and Shuming Shi and Ruifeng Xu},
  title = {{MCPG: A Flexible Multi-Level Controllable Framework for Unsupervised Paraphrase Generation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=_lv7pddT9c},
}
Figure