Solving Hard Analogy Questions with Relation Embedding Chains

Nitesh Kumar, Steven Schockaert

Abstract

Modelling how concepts are related is a central topic in Lexical Semantics. A common strategy is to rely on knowledge graphs (KGs) such as ConceptNet, and to model the relation between two concepts as a set of paths. However, KGs are limited to a fixed set of relation types, and they are incomplete and often noisy. Another strategy is to distill relation embeddings from a fine-tuned language model. However, this is less suitable for words that are only indirectly related and it does not readily allow us to incorporate structured domain knowledge. In this paper, we aim to combine the best of both worlds. We model relations as paths but associate their edges with relation embeddings. The paths are obtained by first identifying suitable intermediate words and then selecting those words for which informative relation embeddings can be obtained. We empirically show that our proposed representations are useful for solving hard analogy questions.

Introduction

Word analogies, such as word:language :: note:music, serve as a critical benchmark for evaluating a model's grasp of lexical semantics. Traditionally, researchers have approached this problem using two distinct strategies: Knowledge Graphs (KGs) and Relation Embeddings. KGs like ConceptNet offer interpretable paths between concepts but are often constrained by a fixed set of relations and inherent incompleteness. Conversely, relation embeddings (e.g., RelBERT) provide nuanced vector representations but often struggle with indirect relationships or "hard" analogies that require multi-step reasoning.

This research, presented at EMNLP 2023, bridges this gap by introducing "Relation Embedding Chains." This hybrid framework models indirect relations as paths through intermediate concepts, where each edge is represented by a high-dimensional embedding rather than a simple discrete label.

Methodology

The proposed framework relies on three core components to handle the complexity of word analogies.

Informativeness Classifier

To determine when a direct relationship is insufficient, the authors developed an informativeness classifier. Trained on 11,000 examples generated by GPT-4, this classifier assigns an "informativeness score" ($inf$) to word pairs. This score acts as a proxy for difficulty; if a direct RelBERT embedding is deemed uninformative, the system pivots to a chain-based approach.

Path Generation

The model identifies intermediate concepts ($x$) to connect two words ($a$ and $b$). To overcome the sparsity of standard KGs, the authors employ Missing Link Prediction and Semantic Smoothing. As illustrated in Figure 1, these techniques allow the model to discover paths that are not explicitly defined in ConceptNet, such as connecting "blandishment" to "entice" through predicted or smoothed links.

Figure 1: Illustration of path generation using ConceptNet links, predicted missing links, and semantic smoothing.

Relation Embedding Chains

Instead of using string labels for edges, the model uses RelBERT embeddings. Two primary methods were tested:

  • Condensed: A neural network aggregates all possible paths into a single vector.
  • Direct: This method compares individual chains of a query pair against candidate answer pairs.

Evaluation and Results

The models were evaluated across seven diverse benchmarks, including the SAT (college entrance level), BATS (morphology), and E-KAR (civil service exams).

Performance Benchmarks

The results, summarized in Figure 2, demonstrate that the hybrid approach—using direct embeddings for "easy" pairs and chains for "hard" ones—achieves competitive performance. Notably, the hybrid model (Cond$_{<0.25}$) achieved a state-of-the-art accuracy of 74.5% on the SAT dataset, outperforming standalone RelBERT and even rivaling some large language models.

Figure 2: Performance comparison across various analogy benchmarks. Bold values indicate the best performance among non-LLM models.

Key Insights

  • Confidence as a Difficulty Proxy: The informativeness score proved to be a reliable predictor of failure for standard models. RelBERT's accuracy plummeted from 71.2% on high-confidence questions to 31.1% on low-confidence ones.
  • The Power of Scaffolding: KGs are most effective when used as a structural starting point, with embeddings "filling in the gaps" to capture subtle semantic nuances.
  • Comparison with LLMs: While GPT-4 remains the top performer in raw accuracy, the proposed embedding chains provide interpretable, reusable vector representations that are more suitable for downstream tasks like information retrieval.

Conclusion and Future Work

Relation Embedding Chains offer a robust solution for "hard" word analogies by combining the structured reasoning of KGs with the flexibility of language model embeddings. While the current implementation focuses on paths of length 2 and involves significant computational overhead, it paves the way for more sophisticated multi-step reasoning in NLP. Future research will explore longer chains and the application of these embeddings to real-world tasks such as legal document retrieval.

Summary updated

Abstract

Modelling how concepts are related is a central topic in Lexical Semantics. A common strategy is to rely on knowledge graphs (KGs) such as ConceptNet, and to model the relation between two concepts as a set of paths. However, KGs are limited to a fixed set of relation types, and they are incomplete and often noisy. Another strategy is to distill relation embeddings from a fine-tuned language model. However, this is less suitable for words that are only indirectly related and it does not readily allow us to incorporate structured domain knowledge. In this paper, we aim to combine the best of both worlds. We model relations as paths but associate their edges with relation embeddings. The paths are obtained by first identifying suitable intermediate words and then selecting those words for which informative relation embeddings can be obtained. We empirically show that our proposed representations are useful for solving hard analogy questions.
BibTeX
@inproceedings{kumar2023solving,
  author = {Nitesh Kumar and Steven Schockaert},
  title = {{Solving Hard Analogy Questions with Relation Embedding Chains}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=qUepn4sdGR},
}
Figure