More than just Frequency? Demasking Unsupervised Hypernymy Prediction Methods

Thomas Bott, Dominik Schlechtweg, Sabine Schulte im WaldeACL 2021

TL;DR

This paper investigates unsupervised methods for predicting hypernym-hyponym relationships between words. It reveals that many popular methods are surprisingly similar to simple frequency-based predictions, highlighting a significant "frequency bias." However, a few methods, particularly those considering word length and second-order entropy, show promise in capturing relationships where frequency-based methods fail.

What Problem Does This Paper Address?

Hypernymy, the "is-a" relationship (e.g., "dog" is a hyponym of "animal"), is a fundamental semantic relation crucial for organizing knowledge and powering various Natural Language Processing (NLP) tasks like taxonomy creation and textual entailment. Predicting this relationship automatically is challenging. While many unsupervised methods exist, they often rely on distributional properties of words (the idea that similar words appear in similar contexts). This paper aims to dissect these methods, specifically focusing on hypernymy prediction (determining which word in a pair is the hypernym), and to understand how their predictions relate to each other and to simpler, frequency-based heuristics. The core problem is to determine if these complex unsupervised methods offer genuine semantic insights beyond simple statistical correlations, particularly frequency.

Methods

The paper compares four unsupervised hypernymy prediction methods:

  • WeedsPrec: An asymmetric precision method measuring the inclusion of features of one word in another.
  • InvCL: An asymmetric precision method that combines feature inclusion and non-inclusion.
  • SLQS Row: A method based on first-order word entropy, using word context features.
  • SLQS Sec: A method based on second-order word entropy, using the median entropy of a word's most associated context words.

Two baselines are used for comparison:

  • Word Length: Predicts the longer word as the hyponym, based on the assumption that more specific concepts might have longer names.
  • Word Frequency: Predicts the less frequent word as the hyponym, assuming more specific concepts appear less often.

The methods are evaluated on their ability to predict which word in a pair is the hyponym and which is the hypernym.

Datasets

The study uses two types of gold standard resources:

  • WordNets: English WordNet (version 3) and German GermaNet (version 11) are used as large-scale lexical taxonomies. Pairs were extracted and filtered for duplicates, autohyponyms, and multiword expressions. A distinction was made between compound and non-compound pairs.
  • Benchmark Datasets:
    • BLESS: Contains relation pairs for 200 English concrete nouns.
  • Lenci and Benotto (2012): A subset of BLESS relation pairs.
  • EVALution: Induced from ConceptNet and WordNet, with filtered pairs for improved consistency.
  • Weeds dataset: Contains word pairs related by hypernymy and co-hyponymy across word classes.

All datasets were filtered to include only noun-noun pairs with a hypernymy relation.

Key Insights and Ideas

  • The Dominance of Frequency: Many unsupervised hypernymy prediction methods (WeedsPrec, InvCL, SLQS Row) show a surprisingly high overlap in their predictions and strongly correlate with simple word frequency baselines. This suggests that their "semantic" predictions are often driven by statistical regularities rather than deeper semantic understanding.
  • Frequency Bias is Pervasive: The study strongly emphasizes the need to check for frequency bias in computational methods. Without this, it's hard to distinguish genuine semantic effects from mere statistical artifacts.
  • Word Length as a Different Signal: The word length baseline, while generally performing worse than frequency, makes distinct predictions. Crucially, it, along with SLQS Sec, can succeed where frequency-based methods fail.
  • SLQS Sec's Niche: The second-order entropy method (SLQS Sec) shows lower overall accuracy but demonstrates an ability to correctly predict hypernymy in cases where frequency-based methods falter, particularly in subsets where hyponym frequencies begin to exceed hypernym frequencies.

Key Results

  • High Correlation: The Simple Matching Coefficient (SMC) analysis shows that WeedsPrec, InvCL, and SLQS Row have very high correlations (often > 0.9) with each other and with word frequency on non-compound pairs in WordNets.
  • Frequency Outperforms: Word frequency is a powerful baseline, often matching or exceeding the performance of the unsupervised methods on benchmark datasets. For example, on BLESS, Word Frequency achieved 62.30% accuracy, while WeedsPrec got 57.52% and InvCL got 63.05%.
  • Compounds are Easy: For compound pairs, word length is an almost perfect predictor, with accuracies above 90% for most methods.
  • SLQS Sec's Complementary Strength: Figure 3 shows that SLQS Sec and Word Length correctly predict a significant proportion of pairs that word frequency misclassifies. For instance, SLQS Sec correctly predicts about 15% of pairs that word frequency gets wrong.

Limitations and Future Work

  • Frequency Bias: The primary limitation identified is the strong influence of frequency on many unsupervised methods, making it difficult to isolate true semantic contributions.
  • Future Work: The authors suggest that future research should focus on developing methods that can disentangle frequency-related effects from genuine semantic insights. They also imply that methods like SLQS Sec, which show complementary strengths, warrant further investigation for their ability to capture nuances missed by frequency-based approaches. The study also highlights the need for careful evaluation across diverse datasets and the importance of considering different types of word pairs (e.g., compounds vs. non-compounds).

Summary updated

BibTeX
@inproceedings{bott2021more,
  author = {Thomas Bott and Dominik Schlechtweg and Sabine Schulte im Walde},
  title = {{More than just Frequency? Demasking Unsupervised Hypernymy Prediction Methods}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}

Loading related content...

Figure