Bidirectional LMs are Better Knowledge Memorizers? A Benchmark for Real-world Knowledge Injection

Yuwei Zhang, Wenhao Yu, Shangbin Feng, Yifan Zhu, Letian Peng, Jayanth Srinivasa, Gaowen Liu, Jingbo Shang

Abstract

Large Language Models (LLMs) are typically static after their initial pre-training, necessitating "knowledge injection" to incorporate recent real-world facts. However, evaluating this process is challenging due to the limitations of existing benchmarks. This paper introduces WIKIDYK, a high-quality, evolving benchmark derived from human-curated Wikipedia "Did You Know..." entries. Our study reveals a significant architectural insight: smaller Bidirectional Language Models (BiLMs), such as T5, are substantially more effective at internalizing and reasoning over new knowledge than the massive Causal Language Models (CLMs) like Llama or Qwen that currently dominate the field. We analyze various training objectives and demonstrate that bidirectional attention provides a fundamental advantage in encoding factual dependencies.

Introduction: The Challenge of Knowledge Injection

As the world changes, LLMs must be updated with new information—from recent news to scientific breakthroughs—without undergoing full retraining. This process, known as knowledge injection, is often hampered by the "Reverse Curse" in modern Causal LMs (CLMs). For instance, a model taught that "A is the father of B" may fail to recognize that "B is the child of A." This paper investigates whether the unidirectional processing of CLMs is a structural barrier to efficient knowledge retention and proposes a more robust evaluation framework.

The WIKIDYK Benchmark

To provide a more realistic assessment of knowledge injection, the authors developed WIKIDYK. Unlike synthetic datasets, WIKIDYK leverages the expert-curated "Did You Know..." section of Wikipedia, ensuring that the facts are both significant and well-verified.

The WIKIDYK pipeline: from daily Wikipedia updates to knowledge injection and multi-dimensional evaluation.

The benchmark consists of 12,290 unique facts and 77,180 questions, designed to test five critical dimensions:

  • Reliability: Direct recall of injected facts.
  • Generality: Understanding of implicit knowledge.
  • Paraphrase: Robustness to variations in question phrasing.
  • Portability: Ability to perform multi-hop reasoning.
  • Locality: Prevention of catastrophic forgetting of original pre-trained knowledge.

Methodology: Comparing Architectures and Objectives

The researchers conducted a comprehensive comparison between Causal LMs (e.g., Llama-2/3, Qwen-2.5, Gemma-3) and Bidirectional LMs (e.g., Flan-T5, RoBERTa). They also explored different training objectives:

  • Next Token Prediction (NTP): The standard objective for CLMs.
  • Synthetic QA: Converting facts into question-answer pairs for fine-tuning.
  • Span Prediction (SP): Masking parts of a sentence and requiring the model to fill them in, a common objective for BiLMs.

Key Results: The BiLM Advantage

The empirical results strongly favor bidirectional architectures. Despite their smaller parameter counts, BiLMs proved to be far more efficient "knowledge repositories" than their causal counterparts.

Comparison of match accuracy across different models and objectives. BiLMs (T5/Flan-T5) consistently outperform CLMs (Llama/Gemma) in reliability and reasoning tasks.

Architectural Superiority

In terms of reliability (direct fact recall), CLMs exhibited a 23% lower accuracy compared to BiLMs. The study suggests that bidirectional attention allows models to capture richer dependencies between entities, whereas the "reduced context visibility" of CLMs makes them less efficient at internalizing relational data.

Memorization Performance

The Flan-T5-770M model achieved a 46.09% match accuracy on reliability tasks, significantly outperforming much larger CLMs which often struggled to reach double-digit accuracy using standard fine-tuning methods. Furthermore, the standard NTP objective used for most LLMs was found to be highly ineffective for knowledge injection, often leading to severe catastrophic forgetting.

Scaling with Modular Frameworks

To address the scaling limits of smaller BiLMs, the authors introduced a modular collaborative framework. This system routes queries to specialized BiLM "knowledge repositories" or a base LLM depending on the scope. This approach improved reliability by up to 29.1% and effectively mitigated the loss of pre-existing knowledge.

Conclusion and Future Work

The findings suggest that for raw knowledge retention and multi-hop reasoning over new facts, model architecture is a more decisive factor than scale. While CLMs remain superior for open-ended generation, BiLMs offer a more effective "hard drive" for factual information. While Retrieval-Augmented Generation (RAG) remains a powerful tool for providing external context, the authors argue that internalizing knowledge is essential for low-latency applications and complex reasoning where context windows are constrained. Future research will focus on hybrid architectures that can dynamically switch between bidirectional and unidirectional modes to combine the best of both worlds.

Summary updated

BibTeX
@inproceedings{zhang2026bidirectional,
  author = {Yuwei Zhang and Wenhao Yu and Shangbin Feng and Yifan Zhu and Letian Peng and Jayanth Srinivasa and Gaowen Liu and Jingbo Shang},
  title = {{Bidirectional LMs are Better Knowledge Memorizers? A Benchmark for Real-world Knowledge Injection}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.771},
}
Figure