Generalized Tuning of Distributional Word Vectors for Monolingual and Cross-Lingual Lexical Entailment

Goran Glavas, Ivan Vulic

Introduction

Standard distributional word embeddings, such as Word2Vec or FastText, effectively capture general semantic similarity but often fail to distinguish between specific logical relations like synonyms, antonyms, and lexical entailment (LE). Lexical entailment is particularly challenging due to its asymmetric nature—for example, a "poodle" is a "dog," but a "dog" is not necessarily a "poodle."

Existing approaches to specialize embeddings for LE typically fall into two categories: joint models, which are computationally expensive to train, and retrofitting models, which only update words present in the training constraints. This paper introduces GLEN (Generalized Lexical ENtailment), a method that learns a global transformation function to specialize the entire vector space, enabling generalization to unseen words and high-performance cross-lingual applications.

Methodology

The core innovation of GLEN is treating specialization as a supervised learning task where a neural function $f$ transforms the distributional space into a specialized space optimized for lexical entailment.

Comparison of standard retrofitting vs. GLEN specialization and cross-lingual projection.

The Transformation Function

Unlike traditional retrofitting, which moves individual points in space, GLEN trains a feed-forward neural network to transform the entire vector space. This allows the model to learn the underlying "shape" of entailment. To handle the inherent asymmetry of LE, GLEN employs a distance metric that combines cosine distance (direction) and Euclidean norm (magnitude). The vector norm is used to encode hierarchical levels, with more general terms having different vector lengths than specific ones.

Training Objectives

The model is trained using 1.5 million LE pairs, 500k synonyms, and 140k antonyms from WordNet and Roget’s Thesaurus. It optimizes four loss functions:

  1. Asymmetric Loss: Enforces specific norm relationships between hyponyms and hypernyms.
  2. Similarity Loss: Pulls synonyms closer together.
  3. Dissimilarity Loss: Pushes antonyms further apart.
  4. Regularization Loss: Ensures specialized vectors do not drift too far from their original distributional meaning.

Evaluation and Results

GLEN was evaluated on the HyperLex dataset for monolingual graded lexical entailment and on binary LE datasets for cross-lingual transfer (English to French, Russian, and Arabic).

Generalization to Unseen Vocabulary

A primary advantage of GLEN is its ability to generalize to words not seen during training. In experiments varying the vocabulary coverage, GLEN demonstrated remarkable robustness. When 0% of the test words were seen during training, GLEN achieved a Spearman correlation of .481, whereas the previous state-of-the-art retrofitting model (LEAR) plummeted to .174.

Performance on HyperLex across different levels of vocabulary coverage during training.

Cross-Lingual Transfer

GLEN enables zero-shot cross-lingual entailment detection. By mapping a target language space (e.g., French) into the English space using standard alignment tools, the English-trained GLEN function can be applied directly. This approach yielded accuracy improvements of over 20% compared to previous specialized cross-lingual models across multiple language pairs, effectively distinguishing between hyponymy (HYPO) and co-hyponymy (COHYP).

Conclusion

GLEN demonstrates that learning a global transformation function is a robust and efficient alternative to word-specific retrofitting. While retrofitting models may slightly outperform GLEN when training data covers nearly the entire vocabulary, GLEN’s ability to generalize to unseen words and transfer across languages makes it significantly more practical for real-world, open-vocabulary NLP applications.

Summary updated

Abstract

Goran Glavaš, Ivan Vulić. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019.
BibTeX
@inproceedings{glavas2019generalized,
  author = {Goran Glavas and Ivan Vulic},
  title = {{Generalized Tuning of Distributional Word Vectors for Monolingual and Cross-Lingual Lexical Entailment}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1476},
}
Figure