Unsupervised Domain Tuning to Improve Word Sense Disambiguation

Judita Preiss, Mark StevensonNAACL 2013

TL;DR

This paper introduces a novel method to improve Word Sense Disambiguation (WSD) by leveraging topic modeling (LDA) to create domain-specific sense distributions. By aligning topics with word senses, their approach enhances WSD performance across various algorithms without requiring modifications to the WSD systems themselves.

What Problem Does This Paper Address?

Word Sense Disambiguation (WSD) is the task of identifying the correct meaning of a word in a given context, which is crucial for many Natural Language Processing (NLP) tasks. A common baseline is the "most frequent sense" (MFS) heuristic, which is surprisingly difficult to beat, especially for unsupervised WSD systems that lack access to annotated training data. While domain knowledge has been shown to be beneficial for WSD, acquiring this knowledge often requires extensive manual annotation for each new domain. This paper addresses the challenge of effectively incorporating domain information into WSD systems in an unsupervised manner, aiming to improve performance without the need for domain-specific manual labeling.

Methods

The core of the paper's contribution lies in its unsupervised domain tuning technique, which uses Latent Dirichlet Allocation (LDA) to infer topic distributions from documents. The authors propose three methods to derive a "sense per topic" distribution, denoted as $p(w, s|t)$:

  1. Sense-based topic model (SBTM): This method builds an LDA topic model directly from documents that have been annotated with word senses by a WSD algorithm. The resulting model captures word-sense combinations within topics.
  2. Linear equations (LinEq): This approach uses the probabilities of senses given a word and document ($p(s|w, d)$) from a WSD algorithm, along with the LDA-derived topic distributions for the document ($p(t|d)$), to form an overdetermined system of linear equations. This system is solved using linear least squares to derive the sense-topic distributions ($p(s|w, t)$).
  3. Topic words (TopicWord): This method leverages the word-topic distributions ($p(w|t)$) generated by LDA. It identifies the most likely words for each topic and then uses these words, along with the target ambiguous word, to build a bag-of-words representation for WSD. This technique is not applicable to WSD algorithms that rely on word order.

Once the "sense per topic" distribution is obtained, it is combined with the LDA-derived topic distribution of a new document ($p(t|d_{new})$) to compute the desired word sense distribution for that document ($p(w, s|d_{new})$). Crucially, after this initial tuning phase, the WSD algorithm itself is no longer needed for disambiguation.

Datasets

The experiments were conducted using a domain-specific WSD corpus from Koeling et al. (2005), which includes manually annotated instances from the FINANCE and SPORTS domains. This corpus contains 100 manually annotated instances for 41 words.

For training the LDA topic models, the authors used a large collection of 80,128 documents randomly selected from the Reuters corpus (Rose et al., 2002). This large corpus was used to build robust topic models, with the authors noting that even a tenth of the Reuters corpus was manageable.

Key Insights and Ideas

  • Domain-specific sense distributions are powerful: The paper strongly supports the idea that word senses are not uniform across domains. By learning topic-specific sense distributions, WSD performance can be significantly boosted.
  • LDA as a domain knowledge extractor: LDA proves to be an effective tool for automatically discovering latent topics within a corpus, which can then be leveraged to infer domain-specific word sense preferences.
  • Unsupervised tuning is feasible and effective: The proposed methods allow for domain adaptation without requiring any manual annotation of the target domain data, making it highly practical.
  • WSD algorithm independence: The core tuning technique can be applied to any WSD algorithm (supervised or unsupervised), making it a versatile enhancement.
  • Sense-based topic models are more informative: The authors found that topic models built from word-sense combinations (SBTM) are more informative and exhibit less overlap compared to models built solely from wordforms.

Key Results

The paper evaluated their techniques using four WSD algorithms: two versions of Personalized PageRank (ppr and w2w), a WordNet similarity-based system (sim), and a Vector Space Model (VSM) based supervised system.

Across all tested WSD algorithms, the proposed domain tuning techniques consistently improved performance compared to their baselines. For instance, with 150 topics:

  • The SBTM model improved the baseline by 3-6 percentage points for the different WSD algorithms.
  • The LinEq method showed even greater improvements, boosting performance by 5-8 percentage points.
  • The TopicWord method also yielded improvements, though its applicability was limited to certain WSD algorithms.

The authors also noted that the SBTM approach, which builds models from word-sense combinations, generally outperformed word-based models in terms of topic informativeness and reduced overlap.

Limitations and Future Work

  • Dependence on WSD algorithm accuracy: The effectiveness of the domain tuning is inherently linked to the accuracy of the WSD algorithm used to generate the initial sense annotations for topic model building.
  • TopicWord limitations: The TopicWord technique is not applicable to WSD algorithms that do not rely on bag-of-words representations (i.e., those sensitive to word order).
  • Similarity algorithm issues: The similarity algorithm (sim) struggled with certain documents, preventing the application of the LinEq technique in some cases.
  • Future Work: The authors suggest exploring the application of their techniques to other domains and potentially investigating how to automatically determine the optimal number of topics. They also mention that while they annotated a portion of the data, it would be possible to restrict the training set further if a reliable distribution of word senses per topic could still be obtained.

Summary updated

BibTeX
@inproceedings{preiss2013unsupervised,
  author = {Judita Preiss and Mark Stevenson},
  title = {{Unsupervised Domain Tuning to Improve Word Sense Disambiguation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2013},
}
Figure