Abstract
Large Language Models (LMs) are increasingly viewed as parametric knowledge bases. However, most evaluations focus on 1-to-1 relations (e.g., "What is the capital of France?"). This study explores the more complex challenge of 1-to-N relations, where a single subject relates to multiple objects (e.g., "Who are the children of Tom?"). We demonstrate that while LMs can memorize individual facts, they fail to reliably enumerate complete sets of objects. Performance degrades significantly as the number of objects ($N$) increases, revealing a fundamental gap between memorization and set-based retrieval.
Introduction
The utility of LMs as knowledge bases depends on their ability to retrieve information accurately. While 1-to-1 relations have a single clear answer, real-world knowledge is often 1-to-N. Retrieving such knowledge requires the model to handle variable cardinality and aggregate discrete facts learned in different contexts.
The authors identify three core challenges for 1-to-N knowledge:
- Variable Cardinality: The number of objects associated with a subject varies.
- Discrete Occurrence: Objects may appear in disparate training contexts, making unified retrieval difficult.
- Aggregation Requirements: Models must output the entire set without missing items (under-recall) or hallucinating new ones (over-recall).
Methodology
The study utilizes sequence-to-sequence (seq2seq) models, specifically BART-base and T5-base, to test two primary training and evaluation schemes:
- Element-valued Supervision (Memorization): The model is trained on individual facts (e.g., "Tom has a child named Emma"). Evaluation involves using beam search to recover individual objects.
- Set-valued Supervision (Enumeration): The model is trained to output the entire list of objects in a single sequence (e.g., "Tom has children named Emma, Lucas, Olivia").
To prevent "catastrophic forgetting" of individual facts when training on sets, the researchers employed a joint training strategy combining both supervision types.
Experimental Setup
To isolate the model's relational reasoning from linguistic noise, the authors constructed a controlled synthetic dataset:
- Relations: Parent-children (from Wikidata) and Director-titles (from IMDb).
- Structure: Subsets were created for 1-to-2, 1-to-3, and 1-to-4 relations.
- Constraints: Entities were limited to four words, and no name served as both a subject and an object to ensure distinguishability.
Key Results and Insights
The experiments revealed several critical findings regarding the limitations of current LM architectures:
Memorization vs. Enumeration
A high capacity for memorization does not translate to successful enumeration. While T5-base could memorize nearly 90% of individual facts, its ability to list all children of a parent was significantly lower. Knowing the individual parts of a set does not automatically enable the model to synthesize the whole.
The "N" Penalty
There is a sharp performance drop as the number of objects ($N$) increases. For 1-to-2 relations, enumeration accuracy hovered between 40-50%. However, for 1-to-4 relations, accuracy plummeted to between 1% and 11%. This suggests a very limited "set capacity" in standard transformer architectures.
Model-Specific Failure Modes
BART and T5 exhibited distinct failure patterns. When faced with difficult enumeration tasks, BART tended to hallucinate incorrect objects. In contrast, T5 tended to repeat correct objects multiple times (duplication), failing to track its own output state.
Conclusion
This research highlights a significant bottleneck in using LMs as comprehensive knowledge bases. While they excel at retrieving isolated facts, they struggle with the logic of set enumeration. Future work may need to incorporate structural aids, such as ordinal numerals (e.g., "The first child is..."), to help models track and retrieve 1-to-N relations more effectively.
Summary updated