Abstract
Large Language Models (LLMs) are often trained on static snapshots of data, leading to challenges with facts that change over time. This paper introduces MULAN, a benchmark designed to investigate whether LLMs internally distinguish between "mutable" facts (e.g., current head of state) and "immutable" ones (e.g., place of birth). Our findings suggest that while LLMs struggle to output correct time-sensitive facts, they encode mutability as a distinct feature in their latent space, making these facts easier to update through in-context learning.
Introduction
LLMs are frequently criticized for being "frozen in time" because they struggle with "time-contingent truths"—facts that drift or mutate. Previous research often attributed this to a lack of temporal awareness. However, we hypothesize that LLMs do encode mutability, even if they cannot always retrieve the most current truth. A key challenge is distinguishing whether poor performance stems from a fact's temporal nature (mutability) or its complexity (cardinality, i.e., having multiple possible objects).
The MULAN Dataset
To address this, we developed the MULAN (Mutability in LANguage models) dataset. It consists of 47,000 subject-relation queries across 35 relations extracted from Wikidata. The dataset balances three categories:
- Immutable-1: One-to-one static facts (e.g., Place of Birth).
- Immutable-N: Static facts with multiple objects (e.g., Languages Spoken).
- Mutable: Time-contingent facts (e.g., Head of Government).
This structure allows for a controlled comparison between mutability and cardinality.
Methods
We analyzed six popular LLMs (including Llama-2 and Falcon) using three approaches:
- Performance and Confidence Probing: Measuring F1 scores and model confidence (probability of the first predicted token) via cloze-style queries.
- Probe Classifiers: Training linear classifiers on internal states and using Minimum Description Length (MDL) to ensure they uncover learned features rather than memorizing labels.
- Knowledge Editing: Testing the ease of updating facts via in-context learning (e.g., "Imagine that [New Fact]. Now, [Query]?").
Key Insights
The Confidence Gap
Models are not just less accurate with mutable facts; they are significantly less confident in their predictions. This suggests that the models "know" these facts are unstable.
Mutability as a Salient Feature
Probe classifiers achieved high accuracy (>85%) in distinguishing mutable from immutable facts, even when controlling for entity frequency. MDL compression results confirmed that mutability is a salient feature in LLM latent spaces, as it is "cheaper" for a probe to learn mutability than a random label.
Mutable Facts are "Softer"
Because mutable facts are represented differently, they are easier to overwrite. LLMs were more likely to successfully update a mutable fact than an immutable one when provided with new information in a prompt. For instance, in Llama-2-Chat, the update success rate for Mutable facts was 0.526, compared to only 0.397 for Immutable-1 facts.
Key Results
Across all models, performance on mutable facts was consistently lower than on immutable ones. F1 scores for Mutable facts (avg. ~24-32%) were roughly half those of Immutable-1 facts (avg. ~50-60%).
Conclusion and Future Work
Our study demonstrates that LLMs internally recognize fact mutability, which influences both their confidence and their susceptibility to updates. Future work should explore whether these findings extend to direct parameter-weight editing (e.g., ROME) and how models learn this mutability from non-chronological training data.
Summary updated