Introduction
Representing the semantic relationships between words is a fundamental challenge in Natural Language Processing. While word embeddings effectively capture individual entity meanings, characterizing the link between a pair—such as the "habitat" relationship between dog and garden—remains difficult. This paper introduces RELATIVE (RELations as LATent dIscourse VEctors), a latent variable model designed to learn "pure" relation vectors by explicitly filtering out irrelevant context.
The Challenge of Relational Noise
Traditional methods for learning relations often rely on co-occurrence statistics within sentences where both words appear. However, these contexts are frequently "noisy." For instance, in the sentence "The pretty dog led her to a lovely garden," the adjectives "pretty" and "lovely" describe the entities themselves rather than their relationship. Standard models that treat all context words with equal weight tend to overfit to specific word pairs, failing to capture the abstract relational concepts necessary for generalization.
Methodology: The RELATIVE Model
RELATIVE treats the relationship between a word pair $(a, b)$ as a latent variable within a generative unigram language model. The core innovation is a Four-Way Split for every context word $w$:
- Relational: $w$ characterizes the relationship between $a$ and $b$.
- Entity A: $w$ relates specifically to word $a$.
- Entity B: $w$ relates specifically to word $b$.
- Background: $w$ is a common noise word.
The model employs the Expectation-Maximization (EM) algorithm to iteratively estimate the probability of a word belonging to the "Relational" category. The final relation vector is a weighted average of context word vectors, prioritized by these relational probabilities. For unseen pairs, the model uses a fallback strategy, interpolating vectors from the nearest neighbors of the target words.
Experimental Setup
The model was trained on a 2018 English Wikipedia dump and evaluated across several benchmarks:
- Lexical Entailment: HyperLex (evaluating "is-a" hierarchies).
- Relation Classification: DiffVec and BLESS.
- Relational Similarity: SemEval-2012 Task 2.
- Text Categorization: 20News, Reuters, BBC, and Ohsumed.
Results and Discussion
Superior Generalization in Lexical Splits
A critical test for relation models is the "lexical split," where training and test sets contain non-overlapping word pairs. RELATIVE excels in this scenario, demonstrating that it learns abstract discourse patterns rather than just memorizing word associations.
As shown in the table above, RELATIVE achieved a Pearson correlation of 53.1 on the HyperLex lexical split, substantially higher than the 47.6 achieved by SeVeN and the 33.9 of Pair2Vec.
Qualitative Purity and Abstract Concepts
Nearest neighbor analysis reveals that RELATIVE captures more abstract relational concepts compared to baselines. For example, for the pair salamander-amphibian, RELATIVE identifies neighbors based on taxonomic structure (e.g., mold-piece, quilt-layer) rather than staying confined to the biological domain.
Downstream Performance
Integrating RELATIVE vectors as features consistently improved performance in text categorization tasks. On the specialized Ohsumed medical dataset, RELATIVE-augmented embeddings outperformed standard FastText embeddings (36.1 vs 34.2 F1), highlighting the value of "relational DNA" in complex classification tasks.
Conclusion
RELATIVE provides a robust, efficient framework for learning distributional relation vectors. By isolating the latent discourse of relationships from entity-specific noise, it achieves state-of-the-art generalization. Future work will address polysemy—handling word pairs with multiple potential relationships—and explore applications in complex reasoning tasks like reading comprehension.
Summary updated