TL;DR
Researchers have developed EDMem, a unified encoder-decoder framework that integrates a massive, latent "Entity Memory" directly into the model architecture. By storing real-world knowledge as dense vectors instead of retrieving external documents, EDMem achieves the accuracy of open-book models with the speed and efficiency of closed-book generation, significantly outperforming standard models like BART on entity-heavy tasks.
Introduction: The Knowledge Retrieval Dilemma
Large Language Models (LLMs) excel at linguistic tasks but often struggle with "entity-intensive" scenarios—answering questions or generating text about specific people, places, or events. Current solutions generally fall into two categories, each with significant trade-offs:
- Closed-book models (e.g., BART, T5): These store knowledge implicitly within their parameters. While fast, they are prone to "hallucinations" and often fail to recall specific, niche facts.
- Open-book/Retrieval models (e.g., RAG, FiD): These search through millions of external documents to find evidence. While accurate, they are computationally expensive and slow, as they must process massive amounts of raw text for every query.
A third niche, Memory-augmented auto-encoders, uses internal "entity libraries." However, these are often limited to selecting entity IDs from a fixed list, lacking the ability to generate flexible, natural language answers or long-form explanations. EDMem bridges this gap by combining the "knowledge" of a memory system with the "generative" power of an encoder-decoder architecture.
Architecture and Methods
EDMem (Encoder-Decoder with Memory) is built upon the Transformer architecture (specifically BART) but incorporates a specialized Entity Memory containing embeddings for 1 million Wikipedia entities.
The model's architecture is divided into "lower" and "upper" layers. When the lower layers identify an entity marker (e.g., [Es]), the hidden state acts as a query to retrieve relevant information from the Entity Memory. This retrieved "knowledge" is then fused back into the hidden states before processing by the upper layers.
Pre-training
The model is pre-trained on Wikipedia using standard Language Modeling and Entity Linking. To ensure the model relies on its memory, the researchers employed "salient span masking," hiding entire entity names and requiring the model to recover them using the memory component.
Decoding Strategies
To ensure accurate generation of entity names during inference, the authors designed three distinct strategies to constrain the output space:
- Free-form: Standard, unconstrained generation.
- Static Entity Linking: The model first predicts relevant entities, then uses a "prefix tree" to force the decoder to generate only those specific names.
- Dynamic Entity Linking: The model predicts relevant entities on-the-fly, constraining the output word-by-word as it generates text.
Key Insights and Results
- Latent Knowledge Efficiency: By storing entity facts as dense vectors (latent representations) rather than raw text, EDMem bypasses the "retrieval-and-reading" bottleneck, accessing the "essence" of an entity without re-reading Wikipedia pages.
- The Decoder "Safety Net": Unlike strict auto-encoders, the encoder-decoder backbone allows EDMem to generate fluent, non-entity text while the memory provides the "hard facts."
- Constraint Prevents Hallucination: Static and dynamic linking methods act as guardrails, forcing the model to choose from its memory "vocabulary" and significantly reducing incorrect name generation.
Performance Highlights
- Accuracy Gains: On TriviaQA, EDMem-static achieved an Exact Match (EM) score of 46.19%, nearly doubling the performance of a standard BART-Large model (25.02%).
- Efficiency: While open-book models like FiD can be more accurate, they are significantly slower. EDMem offers a "sweet spot" of high accuracy with low latency.
- Better Explanations: Human evaluations rated EDMem’s generations as more informative and factually correct than BART’s, particularly in complex tasks like fact-checking explanations (CREAK) or knowledge-grounded dialogue (WoW).
- Unseen Entity Coverage: EDMem showed up to 93% better coverage for entities not explicitly mentioned in the input prompt compared to BART.
Limitations and Future Work
The model's performance is tied to the density of entities in the data; without specific entities from its 1-million-entry memory, it reverts to standard BART behavior. Additionally, the current memory is Wikipedia-based, suggesting a need for domain-specific memories (e.g., medical or legal) for specialized tasks. Future research may focus on scaling the memory further or making it modular for easier updates without full retraining.
Summary updated