A Lexicon of Distributed Noun Representations Constructed by Taxonomic Traversal

Richard F.e. Sutcliffe, Donie O’Sullivan, Fergus MehargCOLING 1994

TL;DR

This paper introduces a method to automatically generate distributed semantic representations for nouns by traversing taxonomic hierarchies in dictionaries. This approach allows for richer, more nuanced word meanings than previously possible, paving the way for more sophisticated natural language processing systems.

What Problem Does This Paper Address?

Capturing the meaning of words in a way that computers can understand is a fundamental challenge in natural language processing (NLP). Traditional methods often struggle to represent the subtle relationships between words and their diverse meanings. The authors aim to address this by creating "distributed representations," where each word's meaning is mapped to a point in a multi-dimensional space. This allows for capturing a wide range of meanings and relationships, making them useful for complex NLP tasks. Previous attempts were limited by the size of available lexicons, but this work seeks to scale up the process significantly.

Methods

The core of the paper is an algorithm that constructs semantic representations by leveraging the structure of dictionaries, specifically the Princeton WordNet. The method works in two main ways:

  • Feature Extraction: It identifies "features" (semantic attributes) from the definitions of words. For example, the definition of "terrier" might yield features like "small," "short-bodied," and "trained to hunt."
  • Taxonomic Traversal: It uses the hierarchical relationships (like "is-a" or hypernyms) within the dictionary. When a word is defined, its definition provides direct features. However, by following the "is-a" links upwards in the hierarchy, the algorithm can also extract features from the definitions of its parent concepts.

Crucially, the algorithm assigns "centralities" to these features. Features derived from closer definitions in the hierarchy receive higher centralities, reflecting their stronger contribution to the word's meaning. This process is repeated, with centralities decreasing at each level of the hierarchy, until the top is reached or the centrality drops to zero. Finally, these feature-centrality pairs are normalized to create a vector representation.

Datasets

The primary dataset used is the Princeton WordNet. The authors specifically highlight its suitability due to its explicit representation of taxonomic links (hypernyms and hyponyms), which simplifies the extraction process. They also mention using the Merriam Webster Compact Electronic Dictionary and the Irish-Irish An Foclóir Beag in previous work.

Key Insights and Ideas

  • Leveraging Taxonomy is Key: The explicit taxonomic structure of WordNet is a major advantage, allowing for a systematic way to gather features from related concepts. This contrasts with traditional dictionaries where such relationships often need to be inferred.
  • Centrality Matters: Assigning decreasing "centrality" to features as you move up the taxonomic hierarchy is an intuitive and effective way to weight the importance of different semantic attributes. Features closer to the word's direct definition are more influential.
  • Distributed Representations Capture Nuance: The resulting representations are not just simple labels but vectors that can capture subtle semantic similarities and differences between words. This is demonstrated by the quantitative results showing how well related words cluster together.
  • Robustness to Errors: The paper suggests that errors in individual features are less detrimental than errors in the taxonomic structure itself, implying a degree of robustness in the method.

Key Results

The authors present preliminary experimental results demonstrating the effectiveness of their approach:

  • Semantic Similarity: The generated representations allow for calculating semantic similarity using dot products.
  • Words within the same category (e.g., "cars") showed high similarity scores (average 0.80).
  • Words from closely related categories (e.g., "flowers" and "trees") showed moderate similarity (average 0.40).
  • Words from distant categories (e.g., "cars" and "dogs") showed low similarity (average 0.10).
  • Lexicon Size: The algorithm was successfully applied to generate representations for all 71,000 nouns in the Princeton WordNet, a significant increase in scale compared to previous work.
  • Representation Characteristics: The generated representations vary in complexity, with an average of 39 features per word, ranging from 17 to 76 features.

Limitations and Future Work

  • Error Sources: The authors acknowledge that inaccuracies can arise from errors in word sense disambiguation and incorrect syntactic categories within the source dictionary. They are working on eliminating these.
  • Objective Measurement: They are also developing objective methods for large-scale performance measurement of the generated lexicon.
  • Feature Selection: While the method extracts features from glosses, the paper notes that they currently select only the first available hypernymic link if multiple exist. Future work could explore more sophisticated ways of handling multiple links or feature selection.
  • Beyond Nouns: The current work focuses on nouns. Extending this methodology to other parts of speech would be a natural next step.

Summary updated

BibTeX
@inproceedings{sutcliffe1994lexicon,
  author = {Richard F.E. Sutcliffe and Donie O’Sullivan and Fergus Meharg},
  title = {{A Lexicon of Distributed Noun Representations Constructed by Taxonomic Traversal}},
  booktitle = {International Conference on Computational Linguistics},
  year = {1994},
}
Figure