Relational Word Embeddings

José Camacho-Collados, Luis Espinosa Anke, Steven Schockaert

Abstract

While word embeddings have been shown to implicitly encode various forms of attributional knowledge, the extent to which they capture relational information is far more limited. In previous work, this limitation has been addressed by incorporating relational knowledge from external knowledge bases when learning the word embedding. Such strategies may not be optimal, however, as they are limited by the coverage of available resources and conflate similarity with other forms of relatedness. As an alternative, in this paper we propose to encode relational knowledge in a separate word embedding, which is aimed to be complementary to a given standard word embedding. This relational word embedding is still learned from co-occurrence statistics, and can thus be used even when no external knowledge base is available. Our analysis shows that relational word vectors do indeed capture information that is complementary to what is encoded in standard word embeddings.

Introduction

In Natural Language Processing (NLP), standard word embeddings like Word2Vec or GloVe are fundamental tools. They excel at capturing attributional similarity—for example, recognizing that Berlin and Moscow are both capital cities. However, they often struggle with specific relational nuances. A standard model may find it difficult to encode the specific "capital-of" relationship between Berlin and Germany without implying that Moscow is also the capital of Germany due to their high similarity.

Existing approaches to this problem typically rely on structured Knowledge Bases (KBs) like WordNet, which are limited by coverage and often handle only discrete, symbolic relations. Alternatively, "relation vectors" for word pairs can be learned, but these are computationally expensive and difficult to integrate into standard pipelines. This paper introduces Relational Word Embeddings (RWE), an unsupervised method that learns individual word vectors specifically designed to encode relational potential from text co-occurrences.

Methodology

The RWE approach follows a two-step unsupervised process:

  1. Learning Relation Vectors ($r_{wv}$): For related words $w$ and $v$ (identified via Pointwise Mutual Information), the model identifies "bridge" words that appear between them in a large corpus (e.g., Wikipedia). The embeddings of these bridge words are averaged to create a temporary "relation vector" $r_{wv}$ describing their connection.
  2. Inducing Relational Word Vectors ($e_w$): A neural network with one hidden layer is trained to predict the bridge relation vector $r_{wv}$ from the relational embeddings $e_w$ and $e_v$.

To ensure the model is symmetric (i.e., the relation between $w$ and $v$ is the same as between $v$ and $w$), the input to the network is the concatenation of the sum $(e_w + e_v)$ and the element-wise product $(e_w \odot e_v)$ of the word vectors.

The RWE architecture uses a neural network to predict the relation vector $r_{wv}$ from a symmetric combination of individual word embeddings $e_w$ and $e_v$.

The resulting vectors $e_w$ are intended to be concatenated with standard embeddings, providing a complementary "relational layer" to the word's representation.

Evaluation and Results

The model was trained on a 2018 English Wikipedia dump and evaluated across several benchmarks, including BLESS and DiffVec for relation classification, and McRae Feature Norms for physical/functional attributes.

Relation Classification

RWE demonstrates significant improvements in identifying relationships such as hypernymy and meronymy. On the BLESS dataset, RWE achieved an accuracy of 94.3%, outperforming standard FastText (92.8%) and the more complex Pair2Vec (91.2%). Notably, RWE also outperformed KB-based methods like "Retrofitting" and "Attract-Repel" on this task, despite being entirely unsupervised.

Comparison of RWE against baseline models on the DiffVec and BLESS relation classification benchmarks.

Feature Modeling and Qualitative Analysis

In the McRae Feature Norms task, RWE achieved the highest overall F1-score (55.2), indicating it captures attributional properties better than standard models. Qualitative nearest-neighbor analysis further revealed that RWE clusters words by functional or geometric relationships (e.g., sphere being near rectangle and conic) rather than just general topical similarity.

Conclusion

Relational Word Embeddings offer a computationally efficient and unsupervised way to enhance standard word representations with relational knowledge. By leveraging "bridge" words as a proxy for relations, RWE captures information that is fundamentally complementary to attributional embeddings. Future work includes exploring more sophisticated architectures like Transformers and testing these embeddings in downstream tasks such as Question Answering and Natural Language Inference.

Summary updated

Abstract

While word embeddings have been shown to implicitly encode various forms of attributional knowledge, the extent to which they capture relational information is far more limited. In previous work, this limitation has been addressed by incorporating relational knowledge from external knowledge bases when learning the word embedding. Such strategies may not be optimal, however, as they are limited by the coverage of available resources and conflate similarity with other forms of relatedness. As an alternative, in this paper we propose to encode relational knowledge in a separate word embedding, which is aimed to be complementary to a given standard word embedding. This relational word embedding is still learned from co-occurrence statistics, and can thus be used even when no external knowledge base is available. Our analysis shows that relational word vectors do indeed capture information that is complementary to what is encoded in standard word embeddings.
BibTeX
@inproceedings{camachocollados2019relational,
  author = {José Camacho-Collados and Luis Espinosa Anke and Steven Schockaert},
  title = {{Relational Word Embeddings}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=6bgZifohnp},
}
Figure