Are Embedded Potatoes Still Vegetables? On the Limitations of WordNet Embeddings for Lexical Semantics

Xuyou Cheng, Michael Sejr Schlichtkrull, Guy EmersonEMNLP 2023 · DOI

Introduction

Knowledge Base Embedding (KBE) models aim to represent entities and relations in a continuous vector space. Traditionally, these models are evaluated on their ability to perform link prediction—guessing missing connections in a graph like WordNet. However, this paper argues that link prediction is a flawed proxy for actual lexical "meaning." By evaluating state-of-the-art KBE models on semantic benchmarks, the authors uncover a startling disconnect: models that excel at graph-based pattern matching often fail to capture the fundamental hierarchical and semantic structures of language.

The Performance Paradox

The central finding of this research is a "performance paradox." There exists a strong negative correlation between a model's success in link prediction (measured by Mean Reciprocal Rank, MRR) and its performance on downstream semantic tasks like word similarity, analogy detection, and Part-of-Speech (POS) tagging.

Table 2: Performance comparison on WN18RR and downstream semantic tasks. Note the inverse relationship between MRR and semantic metrics like Spearman ρ and NER f1.

As shown in the results, high-performing Graph Neural Network (GNN) models like rGAT achieve top-tier link prediction scores (49.98 MRR) but perform poorly on semantic tasks like Named Entity Recognition (30.70 f1). Conversely, linguistically-grounded models like FuncE, despite lower link prediction scores (25.87 MRR), dominate semantic benchmarks, achieving the highest word similarity (51.19 Spearman ρ) and NER score (54.54 f1).

Methodology and Models

The study compares several classes of KBE architectures:

  • Translational/Bilinear: Classic models like TransE and DistMult.
  • Hyperbolic: MuRP, designed for tree-like hierarchies.
  • Graph Neural Networks (GNNs): rGAT and KBGAT, which use neighborhood attention mechanisms.
  • Function-based: The authors adapt FuncE, which represents nodes as membership functions (fuzzy regions), naturally accommodating hierarchical containment.

Models were trained on WN18RR and evaluated on gold-standard benchmarks including SimLex999 (similarity), BATS (analogies), and standard NLP datasets for POS tagging (PTB) and NER (CoNLL-2003).

The Transitivity Trap

A key reason for the failure of many KBE models is their inability to handle transitivity—the logical property where if A is a B and B is a C, then A is a C. In WordNet, this is essential for hypernymy (e.g., a potato is a vegetable because it is a solanaceous vegetable).

Table 5: Accuracy of hypernym prediction across multiple hops. Most models fail to maintain accuracy beyond a single hop, indicating a failure to capture transitivity.

The data reveals that while most models perform well at 1-hop prediction, their accuracy plummets at 2-hops. For instance, TransE drops from 84.41% to just 2.21%. In contrast, FuncE maintains a robust 76.65% accuracy at 2-hops, suggesting its architecture inherently captures the hierarchical logic of the lexicon better than its competitors.

Discussion: Architecture vs. Data

The researchers found that simply increasing the amount of training data (e.g., using the expanded WN25 dataset) did not resolve the poor semantic performance of complex models like rGAT. This suggests the issue is architectural. Complex neural encoders may be "overfitting" to the specific connectivity patterns of the sparse WN18RR graph, learning to predict links without learning the underlying semantic principles those links represent.

Conclusion

This paper serves as a critical call to action for the KBE community. Link prediction should no longer be the sole metric for evaluating embedding quality. To create models that truly "understand" language, researchers must prioritize architectures that are theoretically capable of representing logical properties like transitivity and hierarchy. Until then, our "state-of-the-art" models may continue to forget that an embedded potato is, indeed, still a vegetable.

Summary updated

BibTeX
@inproceedings{cheng2023are,
  author = {Xuyou Cheng and Michael Sejr Schlichtkrull and Guy Emerson},
  title = {{Are Embedded Potatoes Still Vegetables? On the Limitations of WordNet Embeddings for Lexical Semantics}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.542},
}
Figure