Automatically Discovering Word Senses

Patrick Pantel, Dekang LinNAACL 2003

TL;DR

This paper introduces Clustering by Committee (CBC), a novel algorithm that automatically discovers word senses from text. Unlike previous methods that produce a mixed list of similar words for polysemous terms, CBC groups words into distinct clusters, each representing a unique sense, thereby enabling more nuanced semantic analysis.

What Problem Does This Paper Address?

Understanding the different meanings (senses) of a word is crucial for many Natural Language Processing (NLP) tasks, such as information retrieval, machine translation, and question-answering. Traditional methods for finding similar words, based on the distributional hypothesis (words appearing in similar contexts are semantically related), often fail for polysemous words (words with multiple meanings). For instance, a list of words similar to "suit" might include both clothing items (jacket, shirt) and legal terms (lawsuit, litigation), failing to distinguish these distinct senses. This paper tackles the challenge of automatically disambiguating these word senses and representing them as distinct clusters.

Methods

The core of the paper is the Clustering by Committee (CBC) algorithm. CBC operates on word representations derived from a large corpus using pointwise mutual information (PMI) between words and their contexts. The algorithm has three main phases:

  1. Top-k Similar Elements: For each word, the algorithm identifies its k most similar words.
  2. Committee Discovery: CBC searches for "committees" – sets of representative elements that form tight clusters (high intra-cluster similarity) and are dissimilar from other clusters (low inter-cluster similarity). This phase aims to find unambiguous representatives for potential word senses.
  3. Sense Assignment: Each word is assigned to its most similar clusters. Crucially, after assigning a word to a sense (cluster), the features associated with that sense are "removed" from the word's representation. This allows the algorithm to then discover less frequent senses of the word and avoid assigning it to duplicate senses.

The feature representation for each word is a vector where each dimension corresponds to a context (e.g., verb-object pairs), and the value is the PMI between the word and that context.

Datasets

The paper mentions using a "large corpus" for training and demonstrates the output of CBC for "54,685 words on the 3GB ACQUAINT corpus." No new benchmarks are introduced; rather, the focus is on demonstrating the algorithm's output on an existing corpus.

Key Insights and Ideas

  • Moving beyond simple similarity: The paper highlights the limitation of existing word similarity methods that produce a mixed bag of senses for polysemous words. CBC's innovation lies in its ability to disentangle these senses into distinct clusters.
  • "Committees" for robust sense representation: The concept of "committees" as unambiguous representatives for a sense is a key idea. By selecting committee members that are not themselves polysemous (like "Illinois" for a "state" cluster), the algorithm aims for cleaner, more distinct sense representations.
  • Iterative sense discovery: The process of removing features associated with an assigned sense before discovering the next one is crucial. This iterative refinement allows the algorithm to uncover multiple, potentially less frequent, senses of a word.

Key Results

The paper demonstrates the output of CBC for several words, including "suit" and "plant." For "suit," the algorithm successfully separates the clothing sense (e.g., "blouse, slack, legging, sweater") from the litigation sense (e.g., "lawsuit, allegation, case, charge"). Similarly, for "plant," it distinguishes between the industrial sense (e.g., "factory, facility, refinery") and the botanical sense (e.g., "shrub, ground cover, perennial, bulb").

On the 3GB ACQUAINT corpus, CBC discovered senses for 54,685 words, identifying 24,497 polysemous words.

Limitations and Future Work

The paper does not explicitly detail limitations or future work in the provided text. However, a potential limitation, as hinted at by the "state" cluster example, is the reliance on identifying unambiguous committee members. If a word's senses are very subtle or if all potential representatives are themselves polysemous, the algorithm might struggle. Future work could involve exploring more sophisticated methods for committee selection or sense disambiguation.

Summary updated

BibTeX
@inproceedings{pantel2003automatically,
  author = {Patrick Pantel and Dekang Lin},
  title = {{Automatically Discovering Word Senses}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2003},
}
Figure