TL;DR
This paper introduces RelBERT, a novel method for extracting fine-grained relational knowledge from pre-trained language models (LMs) by fine-tuning them to produce relation embeddings. RelBERT achieves state-of-the-art results on analogy benchmarks, significantly outperforming much larger LMs, and demonstrates impressive generalization capabilities to unseen relation types.
What Problem Does This Paper Address?
Many applications, from question answering to recommendation systems, require understanding the relationships between concepts and entities. While Knowledge Graphs (KGs) offer structured knowledge, they are often incomplete and have coarse-grained schemas. Large Language Models (LLMs) can implicitly capture this knowledge, but they are inefficient, difficult to control, and their knowledge is not explicitly represented in a usable format for many tasks. This paper addresses the challenge of efficiently and effectively extracting fine-grained relational knowledge from LMs in a way that is more controllable and interpretable than direct prompting.
Methods
The core of RelBERT is fine-tuning a pre-trained masked language model (MLM), specifically RoBERTa, to generate "relation embeddings." This is achieved by:
- Prompting: Feeding word pairs (head, tail) into the LM within carefully crafted sentence templates (e.g., "Today, I finally discovered the relation between [h] and [t]: [h] is the <mask> of [t]").
- Embedding Extraction: Aggregating the contextualized embeddings from the LM's output layer, typically by averaging the embeddings of the <mask> token or all tokens in the prompt.
- Contrastive Fine-tuning: Training the LM using contrastive loss functions (Triplet Loss, InfoNCE, InfoLOOB) to ensure that embeddings of word pairs with similar relations are closer in the embedding space, while unrelated pairs are pushed further apart.
This approach allows the LM to learn to represent the relationship between word pairs as a fixed-length vector.
Datasets
RelBERT was trained and evaluated on a variety of datasets:
- Training Data:
- RelSim: A dataset of fine-grained semantic relations from SemEval 2012 Task 2.
- ConceptNet: A commonsense knowledge graph.
- NELL-One: A knowledge graph compiled from web data.
- T-REX: A knowledge base constructed by aligning Wikipedia and Wikidata.
- Evaluation Benchmarks (Analogy Questions): SAT, U2, U4, BATS, Google, SCAN, NELL, T-REX, ConceptNet.
- Evaluation Benchmarks (Lexical Relation Classification): BLESS, CogALexV, EVALution, K&H+N, ROOT09.
Key Insights and Ideas
- Fine-tuning for Relational Knowledge: Instead of relying on LLMs' implicit knowledge through prompting, RelBERT explicitly fine-tunes an LM to produce vector representations of relations. This offers a more structured and controllable way to access relational knowledge.
- Generalization Beyond Training Data: A key finding is RelBERT's remarkable ability to model relations it was not explicitly trained on. For instance, a model trained only on lexical relations between concepts performed well on relations involving named entities and morphological analogies. This suggests the LM learns underlying relational principles rather than just memorizing examples.
- Efficiency and Performance: RelBERT, built on relatively small LMs (e.g., 140M parameters), significantly outperforms much larger LLMs (e.g., GPT-3 with 175B parameters) on various benchmarks, demonstrating a highly efficient approach to capturing relational knowledge.
- Prompting vs. Fine-tuning: The paper highlights that while LLMs are powerful, fine-tuning them for specific tasks like relation embedding extraction can yield superior results, especially when efficiency and explicit representation are desired.
Key Results
- State-of-the-Art on Analogy Benchmarks: RelBERT achieved state-of-the-art results on multiple analogy question datasets, outperforming previous methods and large LMs like GPT-3 by significant margins (e.g., 17 percentage points on SAT).
- Superiority over LLMs: RelBERT models, even smaller ones, consistently outperformed larger LLMs (e.g., GPT-2, GPT-J, OPT, T5) on analogy tasks, especially in zero-shot settings.
- Strong Lexical Relation Classification: RelBERT also achieved competitive results on supervised lexical relation classification tasks, often outperforming SotA methods.
- Generalization: Models trained on lexical relations showed strong performance on named entity relations and morphological analogies, demonstrating generalization capabilities.
- Efficiency: RelBERT models are significantly smaller and more efficient than the large LLMs they outperform.
Limitations and Future Work
- Prompt Sensitivity: While fine-tuning reduces sensitivity, the choice of prompt template still impacts performance, with very short templates being problematic.
- Dataset Limitations: The primary training dataset, RelSim, does not cover named entities, yet RelBERT generalizes well to them. However, for specific datasets like NELL and T-REX, training on those datasets themselves showed some improvement, suggesting potential for further dataset curation.
- Conversational LMs: The paper notes that conversational LMs like ChatGPT and GPT-4 cannot be directly evaluated using perplexity scores via their APIs, limiting direct comparison in some settings.
- Future Work: The authors suggest RelBERT can be useful for semantic search, ontology alignment, and improving applications that currently rely on commonsense KGs. Further exploration of prompt design and dataset augmentation is also implied.
Summary updated