Gathering and Generating Paraphrases from Twitter with Application to Normalization

Wei Xu, Alan Ritter, Ralph GrishmanBUCC @ ACL 2013 · OpenReview

Introduction

Natural Language Processing (NLP) tools frequently falter when faced with the informal, noisy, and rapidly evolving language of social media. A fundamental challenge is paraphrasing—the ability to recognize or generate different expressions of the same meaning. While paraphrase datasets exist, they are typically derived from formal sources like news articles. When applied to Twitter, these news-trained models fail to account for the idiosyncratic nature of social media text, such as typos, slang, and context-specific abbreviations (e.g., "bb" for "bigbang" or "tix" for "tickets").

This paper presents a novel approach to automatically extracting a massive dataset of paraphrases from Twitter by leveraging the platform's inherent redundancy. By treating paraphrasing as a monolingual machine translation problem, the authors develop a system that significantly outperforms existing models on social media text and proves highly effective for text normalization.

Data Collection: Exploiting Social Media Redundancy

The core innovation lies in the "Event-Driven Alignment" method. When major events occur—such as elections or product launches—thousands of users tweet similar information using diverse phrasing. The authors group tweets that mention the same Named Entity and Date, using a statistical G-test to identify significant events.

Example tweets grouped by entity and date to form a parallel corpus.

Within these event clusters, pairs of tweets with high lexical overlap (Jaccard similarity $\ge$ 0.5) are identified to create a monolingual parallel corpus. This process yielded a dataset of approximately 4 million sentence pairs from 3 million tweets gathered over a year.

Methodology

The authors utilize Statistical Machine Translation (SMT) tools, specifically GIZA++ and Moses, to model the paraphrasing task. To ensure the system generates novel paraphrases rather than simply repeating the input, they introduce a lexical similarity penalty during the decoding phase.

For the task of text normalization—converting noisy tweets into standard English—the system's flexibility is demonstrated by swapping the Twitter-based language model for one trained on the New York Times. This effectively biases the SMT engine to "translate" informal social media text into grammatical, standard English. Unlike traditional word-to-word dictionaries, this phrase-based approach can correct grammatical errors and resolve context-dependent ambiguities.

Evaluation and Results

The system was evaluated against two primary baselines: a model trained on Amazon Mechanical Turk video descriptions and the Microsoft Research (MSR) news paraphrase corpus.

BLEU vs. PINC scores comparing the Twitter-trained model against out-of-domain baselines.

As shown in Figure 1, the Twitter-trained model consistently achieved higher BLEU scores (measuring meaning preservation) across various PINC scores (measuring n-gram dissimilarity) compared to the out-of-domain baselines. Human evaluation further confirmed these results, with annotators overwhelmingly preferring the paraphrases generated by the in-domain model.

In normalization tasks, the combined system (Twitter phrase table plus a slang dictionary) achieved a state-of-the-art BLEU score of 77.44, outperforming the dictionary-only baseline of 75.07. This demonstrates that phrase-level information is crucial for handling the complexities of social media language.

Conclusion

This research demonstrates that the redundancy of social media is a powerful resource for building robust NLP models. By automatically harvesting naturally occurring paraphrases, the authors created a system capable of both generating diverse expressions and normalizing noisy text. While the heuristic alignment method introduces some noise, the scale and domain-specificity of the resulting data provide a significant advantage over traditional, formal-text datasets. Future work will focus on refining tweet clustering and pair selection to further improve the quality of the training data.

Summary updated

BibTeX
@inproceedings{xu2013gathering,
  author = {Wei Xu and Alan Ritter and Ralph Grishman},
  title = {{Gathering and Generating Paraphrases from Twitter with Application to Normalization}},
  booktitle = {BUCC @ ACL},
  year = {2013},
  url = {https://openreview.net/forum?id=SxTTXXTCbkU},
}
Figure