TL;DR
Large language models (LLMs) struggle to keep their knowledge up-to-date, leading to outdated information. This paper introduces WikiFactDiff, a novel dataset that captures the evolution of factual knowledge over time, enabling more realistic evaluation of LLM update methods. They also provide an evaluation of existing update algorithms on this new benchmark.
What Problem Does This Paper Address?
LLMs are trained on data up to a certain point in time, meaning they are inherently "out of date" with respect to events or facts that emerge after their training cutoff. This "knowledge decay" is a significant problem for real-world applications where accuracy and timeliness are crucial. While methods exist to update LLMs with new facts, they often rely on unrealistic scenarios and lack comprehensive datasets to evaluate their effectiveness across various update types. Existing datasets like CounterFact and zsRE generate facts randomly, failing to reflect how knowledge actually changes in the real world. This paper aims to bridge this gap by creating a realistic, temporally adaptable dataset for evaluating atomic factual knowledge updates.
Methods
The core of this work is the creation of the WikiFactDiff dataset. This dataset is built by comparing two snapshots of the Wikidata knowledge base: one from January 4, 2021 (Told) and another from February 27, 2023 (Tnew). The process involves:
- Preprocessing and Difference: Cleaning and filtering Wikidata dumps to identify changes between the two snapshots, categorizing triples into F⁻ (only in old), F⁺ (only in new), and F⁰ (in both).
- New Entity Detection: Identifying entities that emerged between Told and Tnew.
- Classification Rules: Applying hand-crafted rules to classify triples as 'new', 'obsolete', or 'static' to define various update scenarios (e.g.,
ReplaceObject,Archive,AddObject,AddRelation,AddEntity). - Search for Neighboring Facts: Identifying semantically related triples to assess the "bleedover" effect (how an update to one fact affects others).
- Verbalization: Generating natural language update sentences and cloze tests from the factual triples to make them usable by LLMs and for evaluation.
The paper also presents an evaluation of several existing knowledge update algorithms (FT, FT+L, MEND, ROME, MEMIT) and a new prompting-based method (PROMPT) on a subset of their dataset (WFDrepl), which focuses on the replacement scenario.
Datasets
- WikiFactDiff: The primary contribution, a large dataset of 327K updates reflecting factual knowledge evolution between January 2021 and February 2023, derived from Wikidata. It includes various update scenarios beyond simple replacement.
- WFDrepl: A reduced version of WikiFactDiff, specifically curated for the replacement scenario, to allow comparison with existing benchmarks like CounterFact. It contains 10,412 updates.
- CounterFact & zsRE: Existing datasets used for comparison, noted for their unrealistic, randomly generated updates.
Key Insights and Ideas
- Realism Matters: The paper strongly emphasizes that realistic knowledge evolution (e.g., new entities appearing, facts becoming obsolete) is crucial for evaluating LLM update methods, unlike the random fact generation in prior datasets.
- Temporal Adaptability: WikiFactDiff is designed to be temporally adaptable, meaning it can be updated to reflect knowledge changes over different time periods, making it a long-term resource.
- Beyond Replacement: The dataset supports a richer set of update scenarios (archival, new entity insertion) than previously available, highlighting the limitations of current update algorithms.
- Specificity is Hard: Evaluating "bleedover" (negative impact on unrelated facts) is critical. The paper shows that while methods like ROME and MEND perform well, maintaining specificity remains a challenge, especially for prompting-based approaches.
- Neighboring Facts are Key: The K-nearest-triples approach for detecting bleedover is shown to be more effective than random neighbors, confirming its relevance for evaluating update algorithms.
Key Results
- WikiFactDiff contains 327K updates, categorized into 'new', 'obsolete', and 'static' facts, supporting multiple update scenarios.
- Existing update algorithms (ROME, MEMIT) show strong performance on the WFDrepl subset, achieving high efficacy and generalization.
- However, some methods struggle with specificity (bleedover), particularly FT.
- The PROMPT method, while competitive, exhibits higher bleedover on random neighbors, suggesting that while it's effective for specific updates, its impact on broader knowledge needs careful consideration.
- The paper provides a baseline evaluation of these algorithms on their realistic dataset, offering valuable insights into their strengths and weaknesses.
Limitations and Future Work
- Algorithm Limitations: Current update algorithms are primarily designed for the "replacement" scenario. The paper calls for the development of new algorithms to handle other scenarios like archival and new entity insertion.
- Specificity Measurement: While the K-nearest-triples approach is useful, the paper suggests future work could improve bleedover detection by integrating subject popularity into neighbor selection.
- Prompting Limitations: Prompting-based methods are limited by context window size, unlike parameter-updating methods.
- Future Work: The authors explicitly state that designing algorithms for the challenging new update scenarios is the main direction for future work. They also suggest that the corpus creation pipeline can be adapted for new time periods.
Summary updated