A Latent Variable Model for Learning Distributional Relation Vectors

José Camacho-Collados, Luis Espinosa Anke, Shoaib Jameel, Steven Schockaert

Introduction

Representing the semantic relationships between words is a fundamental challenge in Natural Language Processing. While word embeddings effectively capture individual entity meanings, characterizing the link between a pair—such as the "habitat" relationship between dog and garden—remains difficult. This paper introduces RELATIVE (RELations as LATent dIscourse VEctors), a latent variable model designed to learn "pure" relation vectors by explicitly filtering out irrelevant context.

The Challenge of Relational Noise

Traditional methods for learning relations often rely on co-occurrence statistics within sentences where both words appear. However, these contexts are frequently "noisy." For instance, in the sentence "The pretty dog led her to a lovely garden," the adjectives "pretty" and "lovely" describe the entities themselves rather than their relationship. Standard models that treat all context words with equal weight tend to overfit to specific word pairs, failing to capture the abstract relational concepts necessary for generalization.

Methodology: The RELATIVE Model

RELATIVE treats the relationship between a word pair $(a, b)$ as a latent variable within a generative unigram language model. The core innovation is a Four-Way Split for every context word $w$:

  1. Relational: $w$ characterizes the relationship between $a$ and $b$.
  2. Entity A: $w$ relates specifically to word $a$.
  3. Entity B: $w$ relates specifically to word $b$.
  4. Background: $w$ is a common noise word.

The model employs the Expectation-Maximization (EM) algorithm to iteratively estimate the probability of a word belonging to the "Relational" category. The final relation vector is a weighted average of context word vectors, prioritized by these relational probabilities. For unseen pairs, the model uses a fallback strategy, interpolating vectors from the nearest neighbors of the target words.

Experimental Setup

The model was trained on a 2018 English Wikipedia dump and evaluated across several benchmarks:

  • Lexical Entailment: HyperLex (evaluating "is-a" hierarchies).
  • Relation Classification: DiffVec and BLESS.
  • Relational Similarity: SemEval-2012 Task 2.
  • Text Categorization: 20News, Reuters, BBC, and Ohsumed.

Results and Discussion

Superior Generalization in Lexical Splits

A critical test for relation models is the "lexical split," where training and test sets contain non-overlapping word pairs. RELATIVE excels in this scenario, demonstrating that it learns abstract discourse patterns rather than just memorizing word associations.

Performance comparison on HyperLex lexical and random splits. RELATIVE significantly outperforms unsupervised baselines like SeVeN and Pair2Vec on the challenging lexical split.

As shown in the table above, RELATIVE achieved a Pearson correlation of 53.1 on the HyperLex lexical split, substantially higher than the 47.6 achieved by SeVeN and the 33.9 of Pair2Vec.

Qualitative Purity and Abstract Concepts

Nearest neighbor analysis reveals that RELATIVE captures more abstract relational concepts compared to baselines. For example, for the pair salamander-amphibian, RELATIVE identifies neighbors based on taxonomic structure (e.g., mold-piece, quilt-layer) rather than staying confined to the biological domain.

Nearest neighbor analysis comparing RELATIVE, Pair2Vec, and VecDiff. RELATIVE identifies more abstract relational neighbors.

Downstream Performance

Integrating RELATIVE vectors as features consistently improved performance in text categorization tasks. On the specialized Ohsumed medical dataset, RELATIVE-augmented embeddings outperformed standard FastText embeddings (36.1 vs 34.2 F1), highlighting the value of "relational DNA" in complex classification tasks.

Conclusion

RELATIVE provides a robust, efficient framework for learning distributional relation vectors. By isolating the latent discourse of relationships from entity-specific noise, it achieves state-of-the-art generalization. Future work will address polysemy—handling word pairs with multiple potential relationships—and explore applications in complex reasoning tasks like reading comprehension.

Summary updated

Abstract

Recently a number of unsupervised approaches have been proposed for learning vectors that capture the relationship between two words. Inspired by word embedding models, these approaches rely on co-occurrence statistics that are obtained from sentences in which the two target words appear. However, the number of such sentences is often quite small, and most of the words that occur in them are not relevant for characterizing the considered relationship. As a result, standard co-occurrence statistics typically lead to noisy relation vectors. To address this issue, we propose a latent variable model that aims to explicitly determine what words from the given sentences best characterize the relationship between the two target words. Relation vectors then correspond to the parameters of a simple unigram language model which is estimated from these words.
BibTeX
@inproceedings{camachocollados2019latent,
  author = {José Camacho-Collados and Luis Espinosa Anke and Shoaib Jameel and Steven Schockaert},
  title = {{A Latent Variable Model for Learning Distributional Relation Vectors}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.24963/ijcai.2019/682},
}
Figure