Monolingual Machine Translation for Paraphrase Generation

Chris Quirk, Chris Brockett, William B. DolanEMNLP 2004

Abstract

Paraphrase generation is a fundamental task in natural language processing, yet it has historically been constrained by limited datasets and rigid, template-based methodologies. This paper proposes a novel framework that treats paraphrasing as a monolingual machine translation problem. By utilizing large-scale, automatically clustered news articles as a parallel corpus, the authors apply statistical machine translation (SMT) techniques to learn phrasal correspondences. The resulting system demonstrates high accuracy and broad coverage, significantly outperforming previous state-of-the-art methods in diverse news domains.

Introduction

Paraphrasing—the act of restating a sentence while maintaining its original meaning—is essential for tasks such as summarization, question answering, and information retrieval. At the time of this research, the field faced a significant "data and diversity" bottleneck. Existing approaches typically relied on small, specialized datasets or hand-crafted templates, which limited their applicability to narrow domains. This paper seeks to overcome these limitations by moving toward a scalable, data-driven model that leverages the vast amount of naturally occurring paraphrases found in daily news reporting.

Methodology

The authors frame the paraphrasing task using the Noisy Channel Model, a cornerstone of early statistical machine translation. To identify the most probable paraphrase $T$ for a given source sentence $S$, the model seeks to maximize the product of the translation model and the language model: $P(S|T)P(T)$.

Data Extraction and Alignment

The core innovation lies in the data source. Rather than relying on human-translated parallel texts, the authors crawled the web for news clusters—groups of articles from different sources covering the same events. Within these clusters, sentence pairs likely to be paraphrases were identified using Levenshtein edit distance.

To learn correspondences between words and phrases, the authors employed Giza++, a standard SMT tool. This process identifies which segments of a sentence map to others, even when the wording differs significantly.

Word alignment between two paraphrased sentences using SMT techniques.

Decoding

For generation, the system utilizes a monotone phrasal decoder. This decoder replaces phrases in the source sentence with their learned equivalents based on the alignment model. A deliberate choice was made to keep the decoding "monotone," meaning the original word order is largely preserved. While this prevents structural transformations (like active-to-passive voice), it ensures that the generated output remains grammatically correct and meaning-preserving.

Experimental Setup

The system was trained on a massive corpus of 139,000 non-identical sentence pairs extracted from over 177,000 news articles collected over an eight-month period. Evaluation was conducted on two primary datasets:

  1. B&L59: A benchmark set of 59 sentences from previous research.
  2. Random Set: 141 sentences randomly selected from news clusters to evaluate performance on diverse, "out-of-domain" topics.

Key Results

The performance of the SMT-based Phrasal Replacement (PR) model was evaluated by human raters who judged the "acceptability" of the generated paraphrases.

Performance comparison of the Phrasal Replacement (PR) model against MSA and WordNet baselines.

As shown in the results, the PR model's top-ranked rewrite (PR #1) achieved an acceptance rate of 91.5% on the B&L59 set, significantly outperforming the previous state-of-the-art Multi-Sequence Alignment (MSA) method, which scored 78.0%. Furthermore, the PR model demonstrated remarkable scalability, maintaining an 89.5% acceptance rate on the broader set of 200 sentences. In contrast, simple synonym replacement using WordNet (WN) performed poorly, highlighting the importance of context-aware phrasal learning.

Discussion and Insights

A primary insight of this work is that the web serves as a massive, naturally occurring parallel corpus. The daily news cycle provides high-quality paraphrases at a scale previously unavailable. Additionally, the SMT approach proved robust to noise; despite alignment errors, the combination of phrasal replacement and a strong trigram language model effectively filtered out poor candidates during generation.

Crucially, while previous "ambitious" methods like MSA could perform dramatic rewrites, they often suffered from low coverage (working for only ~12% of sentences). The SMT approach is more "conservative" but achieves 100% coverage, making it a more practical solution for real-world applications.

Limitations and Future Work

The current model's main limitation is its structural rigidity due to the monotone decoder. It cannot perform large-scale syntactic reordering. Future work could involve more sophisticated decoders that allow for non-monotone reordering and the exploration of automated metrics for paraphrase evaluation to replace slow and expensive human judgment.

Summary updated

BibTeX
@inproceedings{quirk2004monolingual,
  author = {Chris Quirk and Chris Brockett and William B. Dolan},
  title = {{Monolingual Machine Translation for Paraphrase Generation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2004},
}
Figure