DocMEdit: Towards Document-Level Model Editing

Li Zeng, Zeming Liu, Chong Feng, Heyan Huang, Yuhang Guo

Introduction

Large Language Models (LLMs) often struggle with "hallucinations" or maintaining up-to-date knowledge. While model editing has emerged as a surgical way to update specific facts without full retraining, research has largely been confined to a "sentence-level" bubble. Current benchmarks typically ask models to output short phrases or single sentences.

In real-world applications, however, we need LLMs to update entire biographies, technical manuals, or Wikipedia entries. This document-level editing presents unique challenges: handling long contexts (1,500+ tokens), managing multiple simultaneous fact updates, and extrapolating new information into a coherent narrative. To address this gap, researchers have introduced DocMEdit, the first benchmark designed for complex, document-scale model editing.

Comparison between sentence-level and document-level model editing.

The DocMEdit Benchmark

The DocMEdit dataset was constructed by comparing Wikipedia dumps from late 2023 and early 2024 to identify factual changes in long-form content. It consists of 37,990 data items containing over 105,000 editing facts. Unlike previous datasets like ZsRE or CounterFact, DocMEdit features:

  • Scale and Complexity: Average context lengths of 1,535 tokens and an average of 2.78 facts per edit.
  • Multi-fact Requirements: Models must often update several related or unrelated points within a single document.

To evaluate performance, the authors proposed several metrics: Document-level Metrics (DR & DE) to measure the overall quality and entity accuracy of the updated document, and Edit-level Metrics (ER & EE) to assess whether the new target facts were successfully injected. Additionally, Locality Metrics (RSE & ESE) ensure that unrelated parts of the document remain unchanged.

Editing Methodologies

The study evaluates two primary families of model editing:

  • Parameter Modification: Methods like Fine-Tuning (FT) and MEMIT directly alter the model's weights to embed new knowledge.
  • Parameter Preservation (RAG-based): Methods like IKE, SKEME, and EREN use Retrieval-Augmented Generation. These keep model weights frozen and provide new facts via the input prompt (in-context learning).

Key Insights and Experimental Results

Experiments across models like Llama2, Mistral, and DeepSeek reveal that document-level editing remains an unsolved problem. Factual accuracy is generally low; even top-performing RAG-based methods like SKEME achieved "Edit-Entity" (EE) scores of only 10-23% on Llama2.

The Context Length Penalty

A critical finding is that performance drops sharply as document length increases. For documents exceeding 2,048 tokens, most current editing methods fail to generate a successfully updated narrative, as evidenced by the decline in Document-Entity (DE) scores.

Performance (DE) across different context lengths for various editing methods.

Hallucination and Fact Density

Manual error analysis showed that 78.4% of editing failures are due to hallucinations. Models often invent "filler" details not present in the source facts. Furthermore, performance degrades as the number of facts per document increases. Interestingly, while RAG-based methods excel with 1-2 facts, simple Fine-Tuning becomes more competitive when juggling 5 or more facts.

Retrieval Bottlenecks

For RAG-based editors, the challenge lies in retrieval. While vector-based retrieval (IKE) struggles with complex facts, entity-based retrieval (SKEME) proved more robust, though still limited by the model's context window and reasoning capabilities.

Conclusion and Future Directions

DocMEdit highlights the sobering reality that current techniques struggle to maintain factual accuracy and narrative consistency in full documents. Future research may need to move beyond modifying Feed-Forward Networks (FFNs) toward attention-focused editing to help models better prioritize new facts. Other promising paths include task decomposition and improving LLM comprehension over long contexts.

Summary updated

BibTeX
@inproceedings{zeng2025docmedit,
  author = {Li Zeng and Zeming Liu and Chong Feng and Heyan Huang and Yuhang Guo},
  title = {{DocMEdit: Towards Document-Level Model Editing}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure