Abstract: Word embedding models such as GloVe rely on co-occurrence statistics from a large corpus to learn vector representations of word meaning. These vectors have proven to capture surprisingly fine-grained semantic and syntactic information. While we may similarly expect that co-occurrence statistics can be used to capture rich information about the relationships between different words, existing approaches for modeling such relationships have mostly relied on manipulating pre-trained word vectors. In this paper, we introduce a novel method which directly learns relation vectors from co-occurrence statistics. To this end, we first introduce a variant of GloVe, in which there is an explicit connection between word vectors and PMI weighted co-occurrence vectors. We then show how relation vectors can be naturally embedded into the resulting vector space.
Introduction
The vector offset hypothesis—famously illustrated by the analogy "King - Man + Woman = Queen"—has long served as the foundation for modeling semantic relations in word embedding spaces like Word2Vec and GloVe. However, this approach assumes that relationships are intrinsically encoded within individual word vectors. This paper argues that many relationships, particularly specific or complex facts, are better captured by the global co-occurrence statistics of words that mediate the relationship. We propose a novel unsupervised method to directly learn relation vectors from these statistics.
Methodology
Our approach extends the GloVe framework to handle word-context-word triples $(i, j, k)$, where $j$ is a context word appearing between or around the pair $(i, k)$.
GloVe Variant and 3-Way PMI
We first establish a direct link between word vectors and Pointwise Mutual Information (PMI) by simplifying the GloVe objective. We then utilize four different generalizations of PMI to three arguments ($SI^1$ through $SI^4$) to quantify the association between a context word $j$ and a pair $(i, k)$.
Learning Relation Vectors
The relation vector $r_{ik}$ is learned by solving a linear least-squares regression problem. By fixing pre-trained context word vectors, we find the vector $r_{ik}$ that best predicts the 3-way PMI values across all context words $j$ in the corpus. This process is computationally efficient and allows the model to capture nuances that simple vector subtraction might miss. The final representation for a pair $(i, k)$ concatenates relation vectors for words appearing between them, before the source, and after the target.
Evaluation and Results
We evaluated our model on standard benchmarks including the Google Analogy test set, DiffVec, and the New York Times (NYT) relation extraction task.
Performance on Analogy Tasks
The proposed relation vectors, particularly the $R^2_{ik}$ variant, consistently outperformed standard baselines such as vector subtraction (Diff), concatenation (Conc), and context averaging (Avg).
As shown in the table above, the $R^2_{ik}$ variant achieves the highest accuracy (92.3%) and F1 score (85.9%), demonstrating the effectiveness of directly modeling the relationship through context aggregation.
Relation Extraction
On the large-scale NYT relation extraction task, our model proved highly competitive with complex neural architectures.
Notably, when paired with a quadratic kernel SVM, the $R^2_{ik}$ model outperformed state-of-the-art neural models like PCNN+ATT at higher recall levels. This suggests that global co-occurrence statistics provide a robust signal for relation extraction that can rival deep learning approaches.
Discussion and Conclusion
The core insight of this work is that relationships are defined by the distribution of words that mediate them. By explicitly counting words in the order $(i, j, k)$, the model captures asymmetric relationships and information not salient in individual word vectors. While the model faces challenges with data sparsity for rare word pairs and high dimensionality in the final representations, it offers a principled, efficient alternative to purely neural relation modeling. By leveraging 3-way co-occurrence statistics, our model achieves superior performance on semantic benchmarks and competitive results on relation extraction tasks, highlighting the power of structured statistical modeling in natural language processing.
Summary updated