Bidirectional LMs are Better Knowledge Memorizers? A Benchmark for Real-world Knowledge Injection

Yuwei Zhang, Wenhao Yu, Shangbin Feng, Yifan Zhu, Letian Peng, Jayanth Srinivasa, Gaowen Liu, Jingbo Shang

TL;DR

Researchers have introduced WIKIDYK, a high-quality, evolving benchmark for testing how well AI models "memorize" new, real-world facts. Their most surprising finding? Smaller Bidirectional Language Models (BiLMs) like T5 are significantly better at internalizing new knowledge than the massive Causal Language Models (CLMs) like Llama or Qwen that dominate the industry today.

What Problem Does This Paper Address?

Large Language Models (LLMs) are usually "frozen" in time after their initial training. To keep them useful, we need to inject new knowledge—like recent news or updated scientific facts—into their parameters. This is called knowledge injection.

However, evaluating this is difficult. Existing benchmarks often use synthetic data or noisy Wikipedia snapshots that don't reflect real-world complexity. Furthermore, modern Causal LMs (which predict the next word in a sequence) suffer from the "Reverse Curse": if you teach them "A is the father of B," they often fail to realize "B is the child of A." This paper investigates whether the very architecture of our most popular models is holding back their ability to learn and recall new information.

Methods

The authors tackle the problem through three main avenues:

  1. Knowledge Injection Techniques: They compared different training objectives, including standard Next Token Prediction (NTP), Synthetic QA (turning facts into question-answer pairs), and Span Prediction (masking a portion of a sentence and asking the model to fill it in).
  2. Architectural Comparison: They pitted Causal LMs (Llama-2/3, Qwen-2.5, Gemma-3) against Bidirectional LMs (Flan-T5, RoBERTa). While CLMs only look at previous words, BiLMs can see the entire context surrounding a fact during training.
  3. Modular Collaborative Framework: To solve the scaling limits of smaller BiLMs, they proposed an ensemble system. This framework uses a "scope classifier" to route questions to specialized BiLM "knowledge repositories" or back to a base LLM if the question is out-of-scope.

Datasets

The centerpiece of the paper is WIKIDYK, a novel benchmark derived from Wikipedia’s “Did You Know...” (DYK) section.

  • Real-world & Expert-Curated: Unlike synthetic datasets, these facts are written and verified by human editors for clarity and significance.
  • Scale: It contains 12,290 unique facts and 77,180 questions.
  • Multi-dimensional Evaluation: It tests five distinct areas: Reliability (direct recall), Generality (implicit facts), Paraphrase (robustness to phrasing), Portability (multi-hop reasoning), and Locality (ensuring the model hasn't forgotten its original training).

Key Insights and Ideas

  • The Architecture Advantage: BiLMs are fundamentally better at encoding facts. Because they use bidirectional attention, they can capture richer dependencies between tokens. CLMs, restricted by causal masking, have "reduced context visibility," making them less efficient at internalizing the relationship between entities.
  • NTP is Not Enough: The standard "Next Token Prediction" objective used to train most LLMs is surprisingly poor for knowledge injection. Models trained this way showed less than 1% accuracy on several recall tasks and suffered from severe catastrophic forgetting of their original knowledge.
  • The "Reverse Curse" is Architectural: The study suggests that the difficulty CLMs have with reversing relationships isn't just a data problem—it's a byproduct of the unidirectional way they process information.
  • Small but Mighty: A 770M parameter Flan-T5 (a BiLM) significantly outperformed much larger CLMs at memorizing facts, proving that for raw knowledge retention, size isn't everything—structure is.

Key Results

  • BiLM Superiority: In terms of reliability (direct fact recall), CLMs exhibited a 23% lower accuracy compared to BiLMs.
  • Memorization Performance: The Flan-T5-770M model achieved a 46.09% match accuracy on reliability tasks, whereas many CLMs struggled to break into double digits using standard fine-tuning.
  • Ensemble Success: The authors' modular framework, which combines multiple BiLMs, improved reliability accuracy by up to 29.1% and helped mitigate the loss of pre-existing knowledge (locality).
  • RAG Comparison: While Retrieval-Augmented Generation (RAG) still provides a performance boost, the authors argue that internalizing knowledge is still critical for low-latency applications and complex reasoning where external context windows are limited.

Limitations and Future Work

The authors acknowledge that while their empirical results are strong, they lack a formal theoretical proof for why bidirectional attention is superior for memorization. Additionally, due to hardware constraints, they could not train a BiLM and a CLM of the exact same scale from scratch under identical conditions; instead, they relied on popular pre-trained versions of each.

Future research directions include exploring hybrid architectures that can adaptively toggle between bidirectional and unidirectional modes, potentially combining the generative prowess of CLMs with the superior "hard drive" capabilities of BiLMs.

Summary updated

Abstract

Despite significant advances in large language models (LLMs), their knowledge memorization capabilities remain underexplored, due to the lack of standardized and high-quality test ground. In this paper, we introduce a novel, real-world and large-scale knowledge injection benchmark that evolves continuously over time without requiring human intervention. Specifically, we propose WikiDYK, which leverages recently-added and human-written facts from Wikipedia's "Did You Know..." entries. These entries are carefully selected by expert Wikipedia editors based on criteria such as verifiability and clarity. Each entry is converted into multiple question-answer pairs spanning diverse task formats from easy cloze prompts to complex multi-hop questions. WikiDYK contains 12,290 facts and 77,180 questions, which is also seamlessly extensible with future updates from Wikipedia editors. Extensive experiments using continued pre-training reveal a surprising insight: despite their prevalence in modern LLMs, Causal Language Models (CLMs) demonstrate significantly weaker knowledge memorization capabilities compared to Bidirectional Language Models (BiLMs), exhibiting a 23% lower accuracy in terms of reliability. To compensate for the smaller scales of current BiLMs, we introduce a modular collaborative framework utilizing ensembles of BiLMs as external knowledge repositories to integrate with LLMs. Experiment shows that our framework further improves the reliability accuracy by up to 29.1%.
BibTeX
@inproceedings{zhang2026bidirectional,
  author = {Yuwei Zhang and Wenhao Yu and Shangbin Feng and Yifan Zhu and Letian Peng and Jayanth Srinivasa and Gaowen Liu and Jingbo Shang},
  title = {{Bidirectional LMs are Better Knowledge Memorizers? A Benchmark for Real-world Knowledge Injection}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.771},
}
Figure