Rethinking the Residual Distribution of Locate-then-Editing Methods in Model Editing

Xiaopeng Li, Shanwen Wang, Shasha Li, Shezheng Song, Bin Ji, Jun Ma, Jie Yu

Large Language Models (LLMs) are typically static once trained, making it difficult to update them with new or corrected information without expensive retraining. Model editing provides a surgical alternative, allowing for specific factual updates. However, current state-of-the-art "locate-then-edit" techniques often struggle with precision and stability. Researchers from the National University of Defence Technology have identified a fundamental flaw in how these methods distribute updates across layers and proposed a streamlined solution: the BLUE (Boundary Layer UpdatE) strategy.

The Pitfalls of Residual Distribution

The dominant paradigm in model editing involves two steps: identifying the "critical layers" that store a specific fact (often via causal tracing) and then applying a mathematical update, or "residual," to those layers. Popular methods like MEMIT and AlphaEdit distribute this residual evenly across a range of layers, operating under the intuition that a distributed update is less disruptive to the model's overall weights.

However, this study reveals that distributing residuals actually introduces significant "weight shift errors." As the update is spread thinner across more layers, its precision diminishes. Theoretical analysis demonstrates that these errors accumulate with the "distribution distance," eventually leading to a degradation of the model's general reasoning capabilities or a failure to successfully integrate the new knowledge. Empirical evidence shows that the contribution of distributed residuals drops off sharply as one moves away from the final layer of the update range.

The BLUE Strategy: Precision through Boundaries

To address these issues, the authors propose BLUE. Unlike existing methods that attempt to update every layer within a critical range, BLUE focuses exclusively on the boundaries.

Comparison between traditional distributed residual updates and the proposed BLUE strategy.

The BLUE process involves:

  1. Identification: Locating the range of critical layers (e.g., layers $L-n+1$ to $L$).
  2. Boundary Updates: Computing and applying direct, mathematically precise updates to only the first and last layers of this range.
  3. Intermediate Preservation: Leaving all intermediate layers untouched.

By eliminating the distribution step, BLUE ensures that the updates remain precise for the specific layers they modify. This "less is more" approach minimizes weight shift errors and reduces the computational overhead associated with multi-layer optimization.

Experimental Results and Stability

The researchers evaluated BLUE across several prominent LLMs, including Llama3-8B, GPT-J 6B, and GPT2-XL, using benchmarks like CounterFact, zsRE, and the GLUE suite.

The results were compelling. BLUE-enhanced methods achieved an average performance improvement of 35.59% over baseline implementations of MEMIT and AlphaEdit. Beyond raw accuracy, BLUE demonstrated superior stability during large-scale sequential editing. While traditional methods often cause a collapse in general language understanding after a few thousand edits, models updated with BLUE maintained their performance on GLUE tasks.

F1 scores on GLUE tasks during 3,000 sequential edits, showing the stability of BLUE-enhanced methods (solid lines) compared to original methods (dashed lines).

Conclusion

The BLUE strategy challenges the long-held assumption that distributing knowledge updates is beneficial for model health. By rethinking the residual distribution process and focusing on boundary layers, BLUE provides a more precise, efficient, and stable framework for model editing. This research suggests that in the complex architecture of LLMs, surgical precision at the boundaries is far more effective than broad, diluted updates.

Summary updated

Abstract

Model editing is a powerful technique for updating the knowledge of Large Language Models (LLMs). Locate-then-edit methods are a popular class of approaches that first identify the critical layers storing knowledge, then compute the residual of the last critical layer based on the edited knowledge, and finally perform multi-layer updates using a least-squares solution by evenly distributing the residual from the first critical layer to the last. Although these methods achieve promising results, they have been shown to degrade the original knowledge of LLMs. We argue that residual distribution leads to this issue. To explore this, we conduct a comprehensive analysis of residual distribution in locate-then-edit methods from both empirical and theoretical perspectives, revealing that residual distribution introduces editing errors, leading to inaccurate edits. To address this issue, we propose the Boundary Layer UpdatE (BLUE) strategy to enhance locate-then-edit methods. Sequential batch editing experiments on three LLMs and two datasets demonstrate that BLUE not only delivers an average performance improvement of 35.59\%, significantly advancing the state of the art in model editing, but also enhances the preservation of LLMs' general capabilities. Our code is available at https://github.com/xpq-tech/BLUE.
BibTeX
@inproceedings{li2025rethinking,
  author = {Xiaopeng Li and Shanwen Wang and Shasha Li and Shezheng Song and Bin Ji and Jun Ma and Jie Yu},
  title = {{Rethinking the Residual Distribution of Locate-then-Editing Methods in Model Editing}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=P9gY05BDkW},
}
Figure