A Unified Encoder-Decoder Framework with Entity Memory

Zhihan Zhang, Wenhao Yu, Chenguang Zhu, Meng Jiang

Abstract

Entities, as important carriers of real-world knowledge, play a key role in many NLP tasks. We focus on incorporating entity knowledge into an encoder-decoder framework for informative text generation. Existing approaches tried to index, retrieve, and read external documents as evidence, but they suffered from a large computational overhead. In this work, we propose an encoder-decoder framework with an entity memory, namely EDMem. The entity knowledge is stored in the memory as latent representations, and the memory is pre-trained on Wikipedia along with encoder-decoder parameters. To precisely generate entity names, we design three decoding methods to constrain entity generation by linking entities in the memory. EDMem is a unified framework that can be used on various entity-intensive question answering and generation tasks. Extensive experimental results show that EDMem outperforms both memory-based auto-encoder models and non-memory encoder-decoder models.

Introduction

Large Language Models (LLMs) have demonstrated remarkable capabilities across various natural language processing tasks. However, they frequently struggle with "entity-intensive" tasks—scenarios requiring precise recall of specific facts about people, places, or events. Currently, the field is divided into two primary paradigms: closed-book models and open-book retrieval models.

Closed-book models, such as BART or T5, store knowledge implicitly within their parameters. While computationally efficient, these models are prone to hallucinations and often fail to recall niche facts. Conversely, open-book models like RAG (Retrieval-Augmented Generation) or FiD (Fusion-in-Decoder) retrieve external documents at inference time. While more accurate, these systems incur significant computational overhead due to the need to index, retrieve, and process large volumes of text for every query.

This paper introduces EDMem, a unified encoder-decoder framework designed to bridge this gap. By integrating a massive internal entity memory, EDMem provides the factual grounding of retrieval systems with the inference speed of closed-book models.

The EDMem Framework

Architecture and Memory Integration

EDMem is built upon the Transformer architecture, specifically utilizing BART-Large as its backbone. The core innovation is a "sandwich" structure for memory access. The model maintains an Entity Memory consisting of dense latent representations for 1 million Wikipedia entities.

To interact with this memory, the model utilizes a specialized query mechanism. During processing, the model identifies potential entity mentions using special tokens: [Es] (entity start) and [Ee] (entity end). When the model encounters an [Es] token, the hidden state from the lower Transformer layers serves as a query to the Entity Memory. The retrieved latent representation is then fused with the model’s internal hidden state and passed to the upper Transformer layers for further reasoning and generation.

Constrained Decoding Methods

To ensure that the retrieved knowledge translates into accurate textual output, the researchers developed three distinct decoding strategies to constrain entity generation:

  1. Free-form Decoding: Standard left-to-right generation without explicit constraints.
  2. Static Entity Linking: The model identifies a set of candidate entities from the memory and utilizes a prefix tree to force the decoder to generate only valid entity names from that subset.
  3. Dynamic Entity Linking: The model predicts entities on-the-fly during the generation process, dynamically constraining subsequent tokens based on these predictions.

Pre-training

The framework is pre-trained on Wikipedia using a dual-objective approach. It combines standard Language Modeling (predicting masked words) with Entity Linking (predicting the specific ID of a masked entity within the memory), ensuring the latent representations are deeply integrated with the generative capabilities of the model.

Experimental Evaluation

Datasets

The authors evaluated EDMem across two broad categories of tasks:

  • Open-domain Question Answering: Natural Questions (NQ), TriviaQA (TQA), and WebQuestions (WQ).
  • Entity-Intensive Generation: MSMARCO (abstractive QA), CREAK (claim verification), ELI5 (long-form QA), and Wizard of Wikipedia (WoW - knowledge-aware dialogue).

Performance and Efficiency

Experimental results demonstrate that EDMem significantly outperforms standard encoder-decoder models. On the TriviaQA dataset, EDMem-static achieved an Exact Match (EM) score of 46.19, a substantial improvement over the baseline BART-Large score of 25.02. It also outperformed specialized memory-based auto-encoders like EaE (43.20).

Beyond accuracy, EDMem offers a massive efficiency advantage over retrieval-based models. In inference tests on the TriviaQA test set, the retrieval-heavy FiD model required approximately 85 minutes to complete the task. In contrast, EDMem-free completed the same task in just 28 seconds, representing a near-instantaneous response time comparable to standard closed-book models.

Human Evaluation

In human evaluations conducted on the ELI5 and WoW datasets, EDMem was consistently rated as more informative and factually grounded than BART. Notably, on the ELI5 dataset, human evaluators occasionally preferred EDMem’s generated responses over the original human-written answers, which were often found to be too brief or informal compared to the model's knowledge-rich output.

Conclusion and Future Work

EDMem represents a significant step forward in creating efficient, factually grounded generative models. By replacing raw text retrieval with latent memory retrieval, it achieves a high degree of accuracy without the latency of traditional RAG systems.

However, certain limitations remain. The model's effectiveness is inherently tied to the density of entities in the input text; tasks with few specific entities may not see significant benefits from the memory overhead. Furthermore, the current memory is specialized for Wikipedia; extending this to domain-specific fields like medicine or law would require specialized pre-training. Finally, addressing the "long-tail" problem—entities not included in the initial 1-million-entry vocabulary—remains a priority for future iterations of memory-augmented frameworks.

Summary updated

Abstract

Entities, as important carriers of real-world knowledge, play a key role in many NLP tasks. We focus on incorporating entity knowledge into an encoder-decoder framework for informative text generation. Existing approaches tried to index, retrieve, and read external documents as evidence, but they suffered from a large computational overhead. In this work, we propose an encoder-decoder framework with an entity memory, namely EDMem. The entity knowledge is stored in the memory as latent representations, and the memory is pre-trained on Wikipedia along with encoder-decoder parameters. To precisely generate entity names, we design three decoding methods to constrain entity generation by linking entities in the memory. EDMem is a unified framework that can be used on various entity-intensive question answering and generation tasks. Extensive experimental results show that EDMem outperforms both memory-based auto-encoder models and non-memory encoder-decoder models.
BibTeX
@inproceedings{zhang2022unified,
  author = {Zhihan Zhang and Wenhao Yu and Chenguang Zhu and Meng Jiang},
  title = {{A Unified Encoder-Decoder Framework with Entity Memory}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=VKtgv6oqrIW},
}
Figure