Abstract
Large Language Models (LLMs) are typically static after their initial pre-training, necessitating "knowledge injection" to incorporate recent real-world facts. However, evaluating this process is challenging due to the limitations of existing benchmarks. This paper introduces WIKIDYK, a high-quality, evolving benchmark derived from human-curated Wikipedia "Did You Know..." entries. Our study reveals a significant architectural insight: smaller Bidirectional Language Models (BiLMs), such as T5, are substantially more effective at internalizing and reasoning over new knowledge than the massive Causal Language Models (CLMs) like Llama or Qwen that currently dominate the field. We analyze various training objectives and demonstrate that bidirectional attention provides a fundamental advantage in encoding factual dependencies.
Introduction: The Challenge of Knowledge Injection
As the world changes, LLMs must be updated with new information—from recent news to scientific breakthroughs—without undergoing full retraining. This process, known as knowledge injection, is often hampered by the "Reverse Curse" in modern Causal LMs (CLMs). For instance, a model taught that "A is the father of B" may fail to recognize that "B is the child of A." This paper investigates whether the unidirectional processing of CLMs is a structural barrier to efficient knowledge retention and proposes a more robust evaluation framework.
The WIKIDYK Benchmark
To provide a more realistic assessment of knowledge injection, the authors developed WIKIDYK. Unlike synthetic datasets, WIKIDYK leverages the expert-curated "Did You Know..." section of Wikipedia, ensuring that the facts are both significant and well-verified.
The benchmark consists of 12,290 unique facts and 77,180 questions, designed to test five critical dimensions:
- Reliability: Direct recall of injected facts.
- Generality: Understanding of implicit knowledge.
- Paraphrase: Robustness to variations in question phrasing.
- Portability: Ability to perform multi-hop reasoning.
- Locality: Prevention of catastrophic forgetting of original pre-trained knowledge.
Methodology: Comparing Architectures and Objectives
The researchers conducted a comprehensive comparison between Causal LMs (e.g., Llama-2/3, Qwen-2.5, Gemma-3) and Bidirectional LMs (e.g., Flan-T5, RoBERTa). They also explored different training objectives:
- Next Token Prediction (NTP): The standard objective for CLMs.
- Synthetic QA: Converting facts into question-answer pairs for fine-tuning.
- Span Prediction (SP): Masking parts of a sentence and requiring the model to fill them in, a common objective for BiLMs.
Key Results: The BiLM Advantage
The empirical results strongly favor bidirectional architectures. Despite their smaller parameter counts, BiLMs proved to be far more efficient "knowledge repositories" than their causal counterparts.
Architectural Superiority
In terms of reliability (direct fact recall), CLMs exhibited a 23% lower accuracy compared to BiLMs. The study suggests that bidirectional attention allows models to capture richer dependencies between entities, whereas the "reduced context visibility" of CLMs makes them less efficient at internalizing relational data.
Memorization Performance
The Flan-T5-770M model achieved a 46.09% match accuracy on reliability tasks, significantly outperforming much larger CLMs which often struggled to reach double-digit accuracy using standard fine-tuning methods. Furthermore, the standard NTP objective used for most LLMs was found to be highly ineffective for knowledge injection, often leading to severe catastrophic forgetting.
Scaling with Modular Frameworks
To address the scaling limits of smaller BiLMs, the authors introduced a modular collaborative framework. This system routes queries to specialized BiLM "knowledge repositories" or a base LLM depending on the scope. This approach improved reliability by up to 29.1% and effectively mitigated the loss of pre-existing knowledge.
Conclusion and Future Work
The findings suggest that for raw knowledge retention and multi-hop reasoning over new facts, model architecture is a more decisive factor than scale. While CLMs remain superior for open-ended generation, BiLMs offer a more effective "hard drive" for factual information. While Retrieval-Augmented Generation (RAG) remains a powerful tool for providing external context, the authors argue that internalizing knowledge is essential for low-latency applications and complex reasoning where context windows are constrained. Future research will focus on hybrid architectures that can dynamically switch between bidirectional and unidirectional modes to combine the best of both worlds.
Summary updated