A Unified Encoder-Decoder Framework with Entity Memory

Zhihan Zhang, Wenhao Yu, Chenguang Zhu, Meng Jiang

TL;DR

Researchers have developed EDMem, a unified encoder-decoder framework that integrates a massive, latent "Entity Memory" directly into the model architecture. By storing real-world knowledge as dense vectors instead of retrieving external documents, EDMem achieves the accuracy of open-book models with the speed and efficiency of closed-book generation, significantly outperforming standard models like BART on entity-heavy tasks.

Introduction: The Knowledge Retrieval Dilemma

Large Language Models (LLMs) excel at linguistic tasks but often struggle with "entity-intensive" scenarios—answering questions or generating text about specific people, places, or events. Current solutions generally fall into two categories, each with significant trade-offs:

  1. Closed-book models (e.g., BART, T5): These store knowledge implicitly within their parameters. While fast, they are prone to "hallucinations" and often fail to recall specific, niche facts.
  2. Open-book/Retrieval models (e.g., RAG, FiD): These search through millions of external documents to find evidence. While accurate, they are computationally expensive and slow, as they must process massive amounts of raw text for every query.

A third niche, Memory-augmented auto-encoders, uses internal "entity libraries." However, these are often limited to selecting entity IDs from a fixed list, lacking the ability to generate flexible, natural language answers or long-form explanations. EDMem bridges this gap by combining the "knowledge" of a memory system with the "generative" power of an encoder-decoder architecture.

Architecture and Methods

EDMem (Encoder-Decoder with Memory) is built upon the Transformer architecture (specifically BART) but incorporates a specialized Entity Memory containing embeddings for 1 million Wikipedia entities.

The EDMem architecture integrates an Entity Memory between the lower and upper Transformer layers.

The model's architecture is divided into "lower" and "upper" layers. When the lower layers identify an entity marker (e.g., [Es]), the hidden state acts as a query to retrieve relevant information from the Entity Memory. This retrieved "knowledge" is then fused back into the hidden states before processing by the upper layers.

Pre-training

The model is pre-trained on Wikipedia using standard Language Modeling and Entity Linking. To ensure the model relies on its memory, the researchers employed "salient span masking," hiding entire entity names and requiring the model to recover them using the memory component.

Decoding Strategies

To ensure accurate generation of entity names during inference, the authors designed three distinct strategies to constrain the output space:

Comparison of Free-Form, Static Entity Linking, and Dynamic Entity Linking decoding strategies.
  1. Free-form: Standard, unconstrained generation.
  2. Static Entity Linking: The model first predicts relevant entities, then uses a "prefix tree" to force the decoder to generate only those specific names.
  3. Dynamic Entity Linking: The model predicts relevant entities on-the-fly, constraining the output word-by-word as it generates text.

Key Insights and Results

  • Latent Knowledge Efficiency: By storing entity facts as dense vectors (latent representations) rather than raw text, EDMem bypasses the "retrieval-and-reading" bottleneck, accessing the "essence" of an entity without re-reading Wikipedia pages.
  • The Decoder "Safety Net": Unlike strict auto-encoders, the encoder-decoder backbone allows EDMem to generate fluent, non-entity text while the memory provides the "hard facts."
  • Constraint Prevents Hallucination: Static and dynamic linking methods act as guardrails, forcing the model to choose from its memory "vocabulary" and significantly reducing incorrect name generation.

Performance Highlights

  • Accuracy Gains: On TriviaQA, EDMem-static achieved an Exact Match (EM) score of 46.19%, nearly doubling the performance of a standard BART-Large model (25.02%).
  • Efficiency: While open-book models like FiD can be more accurate, they are significantly slower. EDMem offers a "sweet spot" of high accuracy with low latency.
  • Better Explanations: Human evaluations rated EDMem’s generations as more informative and factually correct than BART’s, particularly in complex tasks like fact-checking explanations (CREAK) or knowledge-grounded dialogue (WoW).
  • Unseen Entity Coverage: EDMem showed up to 93% better coverage for entities not explicitly mentioned in the input prompt compared to BART.

Limitations and Future Work

The model's performance is tied to the density of entities in the data; without specific entities from its 1-million-entry memory, it reverts to standard BART behavior. Additionally, the current memory is Wikipedia-based, suggesting a need for domain-specific memories (e.g., medical or legal) for specialized tasks. Future research may focus on scaling the memory further or making it modular for easier updates without full retraining.

Summary updated

Abstract

Entities, as important carriers of real-world knowledge, play a key role in many NLP tasks. We focus on incorporating entity knowledge into an encoder-decoder framework for informative text generation. Existing approaches tried to index, retrieve, and read external documents as evidence, but they suffered from a large computational overhead. In this work, we propose an encoder-decoder framework with an entity memory, namely EDMem. The entity knowledge is stored in the memory as latent representations, and the memory is pre-trained on Wikipedia along with encoder-decoder parameters. To precisely generate entity names, we design three decoding methods to constrain entity generation by linking entities in the memory. EDMem is a unified framework that can be used on various entity-intensive question answering and generation tasks. Extensive experimental results show that EDMem outperforms both memory-based auto-encoder models and non-memory encoder-decoder models.
BibTeX
@inproceedings{zhang2022unified,
  author = {Zhihan Zhang and Wenhao Yu and Chenguang Zhu and Meng Jiang},
  title = {{A Unified Encoder-Decoder Framework with Entity Memory}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=VKtgv6oqrIW},
}
Figure