Can LMs Store and Retrieve 1-to-N Relational Knowledge?

Haruki Nagasawa, Benjamin Heinzerling, Kazuma Kokuta, Kentaro InuiACL 2023 · DOI

Abstract

Large Language Models (LMs) are increasingly viewed as parametric knowledge bases. However, most evaluations focus on 1-to-1 relations (e.g., "What is the capital of France?"). This study explores the more complex challenge of 1-to-N relations, where a single subject relates to multiple objects (e.g., "Who are the children of Tom?"). We demonstrate that while LMs can memorize individual facts, they fail to reliably enumerate complete sets of objects. Performance degrades significantly as the number of objects ($N$) increases, revealing a fundamental gap between memorization and set-based retrieval.

Introduction

The utility of LMs as knowledge bases depends on their ability to retrieve information accurately. While 1-to-1 relations have a single clear answer, real-world knowledge is often 1-to-N. Retrieving such knowledge requires the model to handle variable cardinality and aggregate discrete facts learned in different contexts.

Comparison of 1-to-1 and 1-to-N relational knowledge retrieval.

The authors identify three core challenges for 1-to-N knowledge:

  1. Variable Cardinality: The number of objects associated with a subject varies.
  2. Discrete Occurrence: Objects may appear in disparate training contexts, making unified retrieval difficult.
  3. Aggregation Requirements: Models must output the entire set without missing items (under-recall) or hallucinating new ones (over-recall).

Methodology

The study utilizes sequence-to-sequence (seq2seq) models, specifically BART-base and T5-base, to test two primary training and evaluation schemes:

  1. Element-valued Supervision (Memorization): The model is trained on individual facts (e.g., "Tom has a child named Emma"). Evaluation involves using beam search to recover individual objects.
  2. Set-valued Supervision (Enumeration): The model is trained to output the entire list of objects in a single sequence (e.g., "Tom has children named Emma, Lucas, Olivia").
Two supervision strategies: (i) Element-valued supervision for individual fact memorization and (ii) Set-valued supervision for set enumeration.

To prevent "catastrophic forgetting" of individual facts when training on sets, the researchers employed a joint training strategy combining both supervision types.

Experimental Setup

To isolate the model's relational reasoning from linguistic noise, the authors constructed a controlled synthetic dataset:

  • Relations: Parent-children (from Wikidata) and Director-titles (from IMDb).
  • Structure: Subsets were created for 1-to-2, 1-to-3, and 1-to-4 relations.
  • Constraints: Entities were limited to four words, and no name served as both a subject and an object to ensure distinguishability.

Key Results and Insights

The experiments revealed several critical findings regarding the limitations of current LM architectures:

Memorization vs. Enumeration

A high capacity for memorization does not translate to successful enumeration. While T5-base could memorize nearly 90% of individual facts, its ability to list all children of a parent was significantly lower. Knowing the individual parts of a set does not automatically enable the model to synthesize the whole.

The "N" Penalty

There is a sharp performance drop as the number of objects ($N$) increases. For 1-to-2 relations, enumeration accuracy hovered between 40-50%. However, for 1-to-4 relations, accuracy plummeted to between 1% and 11%. This suggests a very limited "set capacity" in standard transformer architectures.

Model-Specific Failure Modes

BART and T5 exhibited distinct failure patterns. When faced with difficult enumeration tasks, BART tended to hallucinate incorrect objects. In contrast, T5 tended to repeat correct objects multiple times (duplication), failing to track its own output state.

Conclusion

This research highlights a significant bottleneck in using LMs as comprehensive knowledge bases. While they excel at retrieving isolated facts, they struggle with the logic of set enumeration. Future work may need to incorporate structural aids, such as ordinal numerals (e.g., "The first child is..."), to help models track and retrieve 1-to-N relations more effectively.

Summary updated

BibTeX
@inproceedings{nagasawa2023can,
  author = {Haruki Nagasawa and Benjamin Heinzerling and Kazuma Kokuta and Kentaro Inui},
  title = {{Can LMs Store and Retrieve 1-to-N Relational Knowledge?}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.acl-srw.22},
}

Loading related content...

Figure