RelBERT: Embedding Relations with Language Models

Asahi Ushio, José Camacho-Collados, Steven SchockaertarXiv 2023 · OpenReview

TL;DR

This paper introduces RelBERT, a novel method for extracting fine-grained relational knowledge from pre-trained language models (LMs) by fine-tuning them to produce relation embeddings. RelBERT achieves state-of-the-art results on analogy benchmarks, significantly outperforming much larger LMs, and demonstrates impressive generalization capabilities to unseen relation types.

What Problem Does This Paper Address?

Many applications, from question answering to recommendation systems, require understanding the relationships between concepts and entities. While Knowledge Graphs (KGs) offer structured knowledge, they are often incomplete and have coarse-grained schemas. Large Language Models (LLMs) can implicitly capture this knowledge, but they are inefficient, difficult to control, and their knowledge is not explicitly represented in a usable format for many tasks. This paper addresses the challenge of efficiently and effectively extracting fine-grained relational knowledge from LMs in a way that is more controllable and interpretable than direct prompting.

Methods

The core of RelBERT is fine-tuning a pre-trained masked language model (MLM), specifically RoBERTa, to generate "relation embeddings." This is achieved by:

  1. Prompting: Feeding word pairs (head, tail) into the LM within carefully crafted sentence templates (e.g., "Today, I finally discovered the relation between [h] and [t]: [h] is the <mask> of [t]").
  2. Embedding Extraction: Aggregating the contextualized embeddings from the LM's output layer, typically by averaging the embeddings of the <mask> token or all tokens in the prompt.
  3. Contrastive Fine-tuning: Training the LM using contrastive loss functions (Triplet Loss, InfoNCE, InfoLOOB) to ensure that embeddings of word pairs with similar relations are closer in the embedding space, while unrelated pairs are pushed further apart.

This approach allows the LM to learn to represent the relationship between word pairs as a fixed-length vector.

Datasets

RelBERT was trained and evaluated on a variety of datasets:

  • Training Data:
    • RelSim: A dataset of fine-grained semantic relations from SemEval 2012 Task 2.
  • ConceptNet: A commonsense knowledge graph.
  • NELL-One: A knowledge graph compiled from web data.
  • T-REX: A knowledge base constructed by aligning Wikipedia and Wikidata.
  • Evaluation Benchmarks (Analogy Questions): SAT, U2, U4, BATS, Google, SCAN, NELL, T-REX, ConceptNet.
  • Evaluation Benchmarks (Lexical Relation Classification): BLESS, CogALexV, EVALution, K&H+N, ROOT09.

Key Insights and Ideas

  • Fine-tuning for Relational Knowledge: Instead of relying on LLMs' implicit knowledge through prompting, RelBERT explicitly fine-tunes an LM to produce vector representations of relations. This offers a more structured and controllable way to access relational knowledge.
  • Generalization Beyond Training Data: A key finding is RelBERT's remarkable ability to model relations it was not explicitly trained on. For instance, a model trained only on lexical relations between concepts performed well on relations involving named entities and morphological analogies. This suggests the LM learns underlying relational principles rather than just memorizing examples.
  • Efficiency and Performance: RelBERT, built on relatively small LMs (e.g., 140M parameters), significantly outperforms much larger LLMs (e.g., GPT-3 with 175B parameters) on various benchmarks, demonstrating a highly efficient approach to capturing relational knowledge.
  • Prompting vs. Fine-tuning: The paper highlights that while LLMs are powerful, fine-tuning them for specific tasks like relation embedding extraction can yield superior results, especially when efficiency and explicit representation are desired.

Key Results

  • State-of-the-Art on Analogy Benchmarks: RelBERT achieved state-of-the-art results on multiple analogy question datasets, outperforming previous methods and large LMs like GPT-3 by significant margins (e.g., 17 percentage points on SAT).
  • Superiority over LLMs: RelBERT models, even smaller ones, consistently outperformed larger LLMs (e.g., GPT-2, GPT-J, OPT, T5) on analogy tasks, especially in zero-shot settings.
  • Strong Lexical Relation Classification: RelBERT also achieved competitive results on supervised lexical relation classification tasks, often outperforming SotA methods.
  • Generalization: Models trained on lexical relations showed strong performance on named entity relations and morphological analogies, demonstrating generalization capabilities.
  • Efficiency: RelBERT models are significantly smaller and more efficient than the large LLMs they outperform.

Limitations and Future Work

  • Prompt Sensitivity: While fine-tuning reduces sensitivity, the choice of prompt template still impacts performance, with very short templates being problematic.
  • Dataset Limitations: The primary training dataset, RelSim, does not cover named entities, yet RelBERT generalizes well to them. However, for specific datasets like NELL and T-REX, training on those datasets themselves showed some improvement, suggesting potential for further dataset curation.
  • Conversational LMs: The paper notes that conversational LMs like ChatGPT and GPT-4 cannot be directly evaluated using perplexity scores via their APIs, limiting direct comparison in some settings.
  • Future Work: The authors suggest RelBERT can be useful for semantic search, ontology alignment, and improving applications that currently rely on commonsense KGs. Further exploration of prompt design and dataset augmentation is also implied.

Summary updated

Abstract

Many applications need access to background knowledge about how different concepts and entities are related. Although Knowledge Graphs (KG) and Large Language Models (LLM) can address this need to some extent, KGs are inevitably incomplete and their relational schema is often too coarse-grained, while LLMs are inefficient and difficult to control. As an alternative, we propose to extract relation embeddings from relatively small language models. In particular, we show that masked language models such as RoBERTa can be straightforwardly fine-tuned for this purpose, using only a small amount of training data. The resulting model, which we call RelBERT, captures relational similarity in a surprisingly fine-grained way, allowing us to set a new state-of-the-art in analogy benchmarks. Crucially, RelBERT is capable of modelling relations that go well beyond what the model has seen during training. For instance, we obtained strong results on relations between named entities with a model that was only trained on lexical relations between concepts, and we observed that RelBERT can recognise morphological analogies despite not being trained on such examples. Overall, we find that RelBERT significantly outperforms strategies based on prompting language models that are several orders of magnitude larger, including recent GPT-based models and open source models.
BibTeX
@misc{ushio2023relbert,
  author = {Asahi Ushio and José Camacho-Collados and Steven Schockaert},
  title = {{RelBERT: Embedding Relations with Language Models}},
  year = {2023},
  eprint = {2310.00299},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.00299},
}
Figure