Two experiments for embedding Wordnet hierarchy into vector spaces

Jean-Philippe Bernardy, Aleksandre MaskharashviliGWC 2019

Abstract

WordNet provides a rich symbolic hierarchy of "is-a" relationships, yet integrating this structure into neural networks remains challenging. This paper evaluates two strategies for embedding WordNet: a mapping model based on fastText vectors and a novel interval disjunction model. Our results demonstrate that while distributional embeddings suffer from low precision in hierarchical tasks, a simple interval-based representation can capture the WordNet hierarchy with over 99.9% recall and perfect precision.

Introduction

WordNet is a comprehensive symbolic database of words and their relationships, such as hyponymy (e.g., a "canary" is a "bird"). While invaluable for linguistics, these symbolic structures are difficult for modern neural networks to process directly. Standard word embeddings like Word2Vec or fastText are optimized for distributional similarity rather than logical hierarchy. In a standard vector space, the distance between concepts is symmetric, failing to capture the inherent asymmetry of hierarchical relations where every bird is an animal, but not every animal is a bird. This research seeks to bridge this gap by encoding hyponymy into feature vectors suitable for machine learning tasks like natural language entailment.

Methodology

The authors propose and test two distinct models for embedding the WordNet hierarchy:

1. The fastText Mapping Model

This approach utilizes existing high-quality fastText word vectors. The model attempts to learn a geometric boundary for every word in WordNet. For a concept like bird, the model defines a region in the vector space; if a sub-concept's vector falls within this boundary, the relationship is confirmed. A "relaxation factor" ($\rho$) is introduced to define inclusion as a proportion of associated vectors falling within the super-concept's boundary.

2. The Interval Disjunction Model

Observing that WordNet is nearly a tree structure (with over 87% of nodes having a single parent), the authors simplify the graph by removing a small fraction of edges (1.7%). They then map this tree to one-dimensional intervals $[low, high]$ derived from a depth-first search. To account for non-tree-like structures where a word has multiple parents, they utilize a "disjunction" of multiple intervals. A word is represented by a set of spans on a number line; if the spans of word A are contained within the spans of word B, A is classified as a hyponym of B.

Interval-based representation of a hierarchy showing how nodes are mapped to numeric spans.

Experimental Setup

The experiments utilized the noun hyponymy hierarchy of WordNet, encompassing over 82,000 nouns. For the mapping model, pre-trained fastText embeddings (trained on 16 billion tokens) were used. Evaluation was performed on the "FTDom" subset, consisting of words present in both the fastText vocabulary and WordNet.

Results and Discussion

The Precision Trap in Distributional Models

The fastText mapping model achieved a respectable recall of 88.8% at a relaxation factor of 0.5. However, it suffered from a critical failure in precision (0.07%). In high-dimensional spaces, the sheer volume of potential "false" relationships (billions of pairs) means that even a low error rate results in an overwhelming number of false positives. The model frequently confused closely related but non-hierarchical terms.

Success of the Interval Model

In contrast, the interval-based approach proved exceptionally robust. By treating WordNet as a tree with a single interval, the model achieved 94.5% recall with 100% precision. By increasing the number of intervals ($n$) to handle complex multi-parent relationships, the performance reached near-perfection.

Recall vs. number of intervals (n) in the disjunction model, showing performance scaling.

As shown in the table above, with $n=10$ intervals, the model captures 99.97% of the hierarchy. This significantly outperforms previous state-of-the-art "order embedding" models, which typically peak in the 90-92% range.

Conclusion

The study highlights a fundamental tension between distributional models, which learn how words are used, and ontological models, which define what words are. While mapping distributional vectors to hierarchies is difficult due to precision issues, the interval disjunction model provides a mathematically elegant and highly accurate way to represent symbolic hierarchies in a format accessible to machine learning. Future work will focus on integrating these interval representations into standard neural architectures and exploring more dynamic linguistic resources.

Summary updated

BibTeX
@inproceedings{bernardy2019two,
  author = {Jean-Philippe Bernardy and Aleksandre Maskharashvili},
  title = {{Two experiments for embedding Wordnet hierarchy into vector spaces}},
  booktitle = {GWC},
  year = {2019},
}
Figure