A Paraphrase and Semantic Similarity Detection System for User Generated Short-Text Content on Microblogs

Kuntal Dey, Ritvik Shrivastava, Saroj KaushikCOLING 2016

TL;DR

This paper introduces a novel machine learning approach for detecting paraphrases and semantic similarity in noisy, short-text social media content like Twitter. By developing a rich set of lexical, syntactic, semantic, and pragmatic features, their system significantly outperforms existing state-of-the-art methods on the SemEval 2015 benchmark, demonstrating its effectiveness across both noisy microblog data and cleaner text corpora.

What Problem Does This Paper Address?

Detecting paraphrases (sentences with the same meaning expressed differently) and semantic similarity (sentences with related meanings) is a crucial task in Natural Language Processing (NLP) with applications in summarization, question answering, and information retrieval. While existing systems achieve high accuracy on "clean" text datasets (like news articles), they struggle with the inherent messiness of user-generated content on platforms like Twitter. This includes abbreviations, misspellings, slang, and a lack of formal structure. The paper aims to bridge this gap by developing a robust system specifically for this challenging domain.

Methods

The authors employ a machine learning approach, specifically using Support Vector Machines (SVMs), to classify tweet pairs. Their core contribution lies in a comprehensive set of features designed to capture various linguistic aspects of the text:

  • Preprocessing: Includes removing topic phrases, normalizing slang and colloquialisms using dictionaries, correcting named entity (NE) boundaries, and cleaning NE tags for consistency.
  • Lexical Features: Character-level and word-level n-gram overlaps (unigrams, bigrams, trigrams), including union, intersection, and difference sizes. They also incorporate stemmed and stopword-removed versions of these features, along with Jaro-Winkler string similarity.
  • Syntactic Features: Focus on Part-of-Speech (POS) agreement, counting matching words with identical POS tags and matching verbs with identical verb tags. They also consider overlap of named entities (NEs) beyond the core topic.
  • Semantic Features: Leverage WordNet to compute word and phrase overlaps based on synonyms and hypernyms. This includes "best word overlap" and "adjective overlap" features, as well as "phrase overlap" features for noun and verb phrases.
  • Pragmatic Features: A boolean feature indicating agreement in subjectivity/objectivity of the tweet pair.

Datasets

The primary dataset used for training and evaluation is the SemEval 2015 Task 1 dataset for paraphrase and semantic similarity in Twitter. This dataset is specifically designed for noisy, short-text content. For comparative analysis and to demonstrate the system's generalizability, the authors also use the Microsoft Paraphrase Corpus, a well-established benchmark for clean-text paraphrase detection.

Key Insights and Ideas

  • Feature Engineering is Key for Noisy Text: The paper strongly suggests that carefully crafted features, rather than just relying on off-the-shelf deep learning models, are crucial for handling the complexities of microblog text.
  • Ubiquity Across Text Types: The proposed feature set, when applied to both noisy Twitter data and clean text corpora, demonstrates a remarkable ability to perform well across different domains, suggesting its robustness.
  • Semantic Features are Highly Beneficial: The inclusion of WordNet-based semantic features significantly boosts performance, highlighting the importance of understanding word meanings beyond simple string matching.
  • Topic Removal is Crucial for Twitter: Explicitly removing topic phrases from tweet pairs is identified as a key factor in improving performance on Twitter data, as it removes noise that doesn't contribute to distinguishing paraphrases.

Key Results

  • On the SemEval 2015 Twitter dataset, their system achieved an F1-score of 0.717 for paraphrase detection (outperforming the state-of-the-art 0.696) and 0.741 for semantic similarity detection (outperforming the state-of-the-art 0.724).
  • When trained on the Microsoft Paraphrase Corpus (clean text), their system achieved an F1-score of 0.824, ranking within the top 10 systems.
  • The paper provides detailed ablation studies (Table 2) showing the impact of individual feature sets, demonstrating that the combination of all proposed features yields the best results.

Limitations and Future Work

  • Deep Learning Potential: The authors acknowledge that deep learning models might offer stronger performance with larger datasets. They suggest that as more extensive noisy short-text paraphrase corpora become available, revisiting deep learning approaches would be beneficial.
  • Cross-Data-Type Adaptation: While their system shows good performance on both clean and noisy text, they note that topic removal, a key feature for Twitter, cannot be applied to the Microsoft Paraphrase corpus. They suggest exploring how to unify the feature set for better cross-dataset adaptation.
  • Pragmatic Features: The subjectivity/objectivity feature was not included in the final Twitter results to maximize F1-score, but showed marginal improvement on the Microsoft Paraphrase dataset. Further exploration of pragmatic features could be beneficial.
  • Open for Exploration: The authors suggest that their feature set could be applied to other types of data and potentially improve system performance in applications like social information spread modeling.

Summary updated

BibTeX
@inproceedings{dey2016paraphrase,
  author = {Kuntal Dey and Ritvik Shrivastava and Saroj Kaushik},
  title = {{A Paraphrase and Semantic Similarity Detection System for User Generated Short-Text Content on Microblogs}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2016},
}
Figure