Specialising Word Vectors for Lexical Entailment

Ivan Vulic, Nikola Mrksic

Abstract

We present LEAR (Lexical Entailment Attract-Repel), a novel post-processing method that transforms any input word vector space to emphasise the asymmetric relation of lexical entailment (LE), also known as the IS-A or hyponymy-hypernymy relation. By injecting external linguistic constraints (e.g., WordNet links) into the initial vector space, the LE specialisation procedure brings true hyponymy-hypernymy pairs closer together in the transformed Euclidean space. The proposed asymmetric distance measure adjusts the norms of word vectors to reflect the actual WordNet-style hierarchy of concepts. Simultaneously, a joint objective enforces semantic similarity using the symmetric cosine distance, yielding a vector space specialised for both lexical relations at once. LEAR specialisation achieves state-of-the-art performance in the tasks of hypernymy directionality, hypernymy detection, and graded lexical entailment, demonstrating the effectiveness and robustness of the proposed asymmetric specialisation model.

Introduction

Standard word embeddings, such as Word2Vec or GloVe, are typically built on the distributional hypothesis: words appearing in similar contexts share similar meanings. While effective for capturing broad semantic similarity, this approach often fails to distinguish between different types of lexical relations. Specifically, it struggles with the asymmetry of lexical entailment (LE). For instance, while "dog" and "animal" are semantically related, the relationship is directional: a dog is an animal, but an animal is not necessarily a dog.

Traditional embeddings rely on cosine similarity, which is inherently symmetric. This makes it difficult for models to differentiate between synonyms (e.g., dog/canine) and hierarchical entailments (e.g., dog/animal). This paper introduces LEAR (Lexical Entailment Attract-Repel), a post-processing method designed to "specialise" word vectors to better handle the directionality and hierarchy of LE.

The LEAR Method

LEAR operates as a post-processing refinement of existing word vectors, treating the initial space as a "black box" and applying external linguistic constraints from resources like WordNet. The core innovation lies in a joint objective function that optimizes for four distinct goals:

  1. Attract Synonyms: Pulls synonymous pairs closer using symmetric cosine distance.
  2. Repel Antonyms: Pushes antonymous pairs further apart.
  3. Regularization: Ensures the specialized vectors retain the useful distributional information from the original space.
  4. Asymmetric LE Specialization: This is the model's primary contribution. It forces lower-level concepts (hyponyms) to have smaller vector norms than higher-level concepts (hypernyms).
Visual representation of the LEAR vector space where vector norms encode hierarchical generality.

By using the L2 norm (vector length) to represent generality, LEAR creates a space where "moving up" the hierarchy corresponds to an increase in vector magnitude. This allows the model to calculate both the "intensity" and "direction" of an IS-A relationship mathematically.

Evaluation and Benchmarks

The model was trained using over 1 million synonym pairs and 1.5 million lexical entailment pairs. Evaluation was conducted across several rigorous benchmarks:

  • BLESS, WBLESS, and BIBLESS: Datasets for testing hypernymy detection and directionality.
  • HyperLex: A gold-standard dataset for graded lexical entailment, measuring the degree to which one concept entails another.
  • SimLex-999 & SimVerb-3500: Used to ensure that hierarchical specialization does not degrade the vectors' ability to represent basic semantic similarity.

Key Results

LEAR consistently outperformed previous state-of-the-art models across all tested tasks. On the HyperLex graded LE task, LEAR significantly improved the Spearman’s correlation from a previous best of 0.540 to 0.686.

Performance of LEAR across different base word vectors on the HyperLex graded lexical entailment task.

In directionality tasks on the BLESS dataset, it achieved an accuracy of 0.96, surpassing the previous baseline of 0.92. Crucially, these improvements were robust across various starting vector spaces (SGNS, GloVe, FastText) and did not come at the cost of performance on standard similarity tasks like SimLex-999. The synergy between symmetric similarity (angles) and asymmetric distance (norms) proved to be the key to this success.

Conclusion and Future Work

LEAR demonstrates that vector norms are an elegant and effective way to encode hierarchical generality in word embeddings. While the current model focuses on noun and verb hierarchies, future work could extend this logic to other parts of speech or different asymmetric relations, such as meronymy (part-whole relationships). Additionally, addressing the "path length" between concepts in a hierarchy remains an open challenge for future iterations of the model.

Summary updated

Abstract

We present LEAR (Lexical Entailment Attract-Repel), a novel post-processing method that transforms any input word vector space to emphasise the asymmetric relation of lexical entailment (LE), also known as the IS-A or hyponymy-hypernymy relation. By injecting external linguistic constraints (e.g., WordNet links) into the initial vector space, the LE specialisation procedure brings true hyponymy-hypernymy pairs closer together in the transformed Euclidean space. The proposed asymmetric distance measure adjusts the norms of word vectors to reflect the actual WordNet-style hierarchy of concepts. Simultaneously, a joint objective enforces semantic similarity using the symmetric cosine distance, yielding a vector space specialised for both lexical relations at once. LEAR specialisation achieves state-of-the-art performance in the tasks of hypernymy directionality, hypernymy detection, and graded lexical entailment, demonstrating the effectiveness and robustness of the proposed asymmetric specialisation model.
BibTeX
@inproceedings{vulic2018specialising,
  author = {Ivan Vulic and Nikola Mrksic},
  title = {{Specialising Word Vectors for Lexical Entailment}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2018},
  url = {https://openreview.net/forum?id=SJlF_XWOWB},
}
Figure