Near-Synonym Choice in an Intelligent Thesaurus

Diana InkpenNAACL 2007

TL;DR

This paper tackles the challenge of automatically selecting the most contextually appropriate near-synonym for a given word, going beyond standard thesauri by ranking suggestions. They propose and evaluate several statistical and machine learning methods, demonstrating that a web-corpus-based approach using Pointwise Mutual Information (PMI) significantly outperforms previous methods and is comparable to supervised learning, without requiring extensive training data.

What Problem Does This Paper Address?

When writing, we often need to find alternative words to avoid repetition or to improve clarity. Standard thesauri provide lists of similar words (near-synonyms), but they don't tell us which word is best suited for a particular sentence. This paper addresses the problem of automatically ranking these near-synonyms based on their semantic coherence with the surrounding text. The core difficulty lies in distinguishing between near-synonyms that have very subtle differences in meaning and usage, which can be hard to capture even for human judges.

Methods

The authors explore three main approaches:

  • Statistical Method (PMI-based): This is their primary proposed method. It leverages the web as a massive corpus to compute Pointwise Mutual Information (PMI) scores between candidate near-synonyms and the words in the surrounding context. PMI quantifies how much more likely two words are to co-occur than by chance. They use a window of words around the target word to calculate this score. They also explore variations using document counts (PMI-IR) instead of raw word counts.
  • Anti-collocations Method: This method uses a pre-compiled knowledge base of "anti-collocations" – word combinations that native speakers would typically avoid. It ranks near-synonyms that don't form anti-collocations with the context higher.
  • Supervised Learning Method: This approach treats the problem as a classification task. They train classifiers (like Decision Trees, Naïve Bayes) using features derived from PMI scores and word presence in the context. The training data is generated by creating lexical gaps in sentences and using the original word as the correct label.

Datasets

The primary dataset used is a large web corpus (approximately one terabyte) collected by the Waterloo MultiText system. For evaluation, they utilize existing datasets derived from the Wall Street Journal (WSJ) corpus (1987 and 1989) and the British National Corpus (BNC). These datasets consist of sentences with a single word replaced by a "lexical gap," and the candidate near-synonyms for that word form the solution set.

Key Insights and Ideas

  • The Web as a Goldmine for Context: Using the vastness of the web as a corpus proves highly effective for capturing subtle semantic nuances of near-synonyms, compensating for the inherent noise in web data.
  • PMI is a Powerful Signal: Pointwise Mutual Information, when applied to a large web corpus, is a strong indicator of contextual suitability for near-synonyms.
  • Unsupervised Simplicity Wins: The unsupervised PMI-based method achieves comparable performance to supervised methods but has a significant advantage: it doesn't require specific training for each group of near-synonyms, making it more scalable.
  • Human Judgment is Difficult: Even human judges struggle with this task, indicating the inherent complexity and the need for sophisticated computational approaches.
  • Context Window Matters: The size of the context window (number of words considered to the left and right of the gap) and the query frame size for PMI calculation are important parameters that need tuning.

Key Results

  • The proposed statistical PMI-based method significantly outperforms a previous method by Edmonds (1997) by an average of 10 percentage points.
  • It also performs considerably better than a simple baseline that always chooses the most frequent near-synonym, with an average improvement of 22 percentage points.
  • The unsupervised PMI-based method achieves accuracies in the range of 66-75% for the best choice.
  • The accuracy for the top two choices is high, around 90%, suggesting that even if the absolute best choice isn't selected first, the user is likely to find a suitable alternative within the top suggestions.
  • The supervised learning methods perform comparably to the unsupervised PMI method, but the unsupervised approach is preferred due to its lack of training requirements.
  • The anti-collocation method shows improvement over the baseline but is less effective than the PMI-based method.

Limitations and Future Work

  • Underestimated Accuracy: The evaluation method, which considers only the original word as the "correct" solution, likely underestimates the true performance, as other near-synonyms might also be contextually appropriate.
  • Future Work: The authors plan to incorporate word sense disambiguation to handle target words with multiple senses, potentially triggering different sets of near-synonyms. They also aim to explore different inventories of near-synonyms and investigate the application of this technology in Natural Language Generation (NLG) and Machine Translation. They suggest combining supervised and unsupervised methods, using supervised learning when confidence is high and unsupervised otherwise.

Summary updated

BibTeX
@inproceedings{inkpen2007near,
  author = {Diana Inkpen},
  title = {{Near-Synonym Choice in an Intelligent Thesaurus}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2007},
}
Figure