More Edits, More Stable: Understanding the Lifelong Normalization in Sequential Model Editing

Xin Ma, Wei Chen, Qi Liu, Derong Xu, Zhi Zheng, Tong Xu, Enhong Chen

Updating a Large Language Model (LLM) with new facts is akin to performing open-heart surgery while the patient is running a marathon. In the field of Lifelong Model Editing (LME), researchers attempt to continuously inject new knowledge into a model without breaking its existing capabilities. Historically, this process has been notoriously fragile, often leading to catastrophic forgetting or total model collapse.

A recent paper, "More Edits, More Stable: Understanding the Lifelong Normalization in Sequential Model Editing," pulls back the curtain on why some editing methods succeed where others fail. By providing the first theoretical explanation for the success of Lifelong Normalization (LN), the authors introduce a highly scalable method capable of handling millions of sequential updates.

The Challenge of Continuous Updates

LLMs are static snapshots of the world at the time they were trained. When facts change—such as a new head of state being elected—the model's knowledge must be updated. Because retraining from scratch is prohibitively expensive, researchers use "model editing" to surgically alter specific parameters.

In a lifelong setting, where edits occur sequentially over time, two primary failure modes typically emerge:

  1. Catastrophic Forgetting: Updating the model with a new fact accidentally overwrites previously learned facts.
  2. Model Collapse: Small errors in each update accumulate, causing the model’s internal representations to drift so far that it loses its general reasoning and linguistic abilities.

Empirically, the most successful editors mitigate these issues using a technique called Lifelong Normalization (LN), which normalizes parameter gradients using running statistics. However, LN has long been treated as a "black box." While practitioners knew it worked, the underlying mechanisms driving its success remained poorly understood.

Demystifying Lifelong Normalization

The authors frame LME as a Bayesian recursive tracking problem. They model the shifting distribution of "value gradients" (the directional shifts parameters must make to learn a new fact) and use a Normal-Inverse-Wishart prior to track these shifts online.

Their theoretical analysis reveals a self-reinforcing stability loop at the heart of LN:

  • Calibrated Updates: Accurate tracking of gradients leads to highly calibrated parameter adjustments.
  • Bounded Norms and Orthogonality: These updates are mathematically constrained so they aren't too large (bounded norms) and don't interfere with each other (asymptotic orthogonality).
  • The Positive Cumulative Effect: Because the updates are stable, the model's internal distribution drifts slowly. Counter-intuitively, this means that early edits actually stabilize the environment, acting as "virtual samples" that make gradient estimation more accurate for future edits.

Introducing StableEdit

Building on these theoretical insights, the authors propose STABLEEDIT, a novel method that strengthens the stability loop to improve upon existing state-of-the-art editors. It introduces two key mechanisms:

  1. Warm-up Stage: STABLEEDIT uses a small set of initial edits to "prime" the running statistics before the main sequence begins. This allows the model to skip the high-error phase typically associated with a "cold start."
  2. Full Whitening: Instead of simply scaling by variance (diagonal normalization), STABLEEDIT utilizes full covariance whitening. This acts as an adaptive preconditioner that decorrelates gradients, amplifying weak signals and damping noise to prevent the model from overfitting to dominant data directions.

Empirical Performance and Scalability

The researchers evaluated STABLEEDIT across a rigorous gauntlet of benchmarks, ranging from standard scale datasets (ZsRE, FEVER) to ultra-large scale environments (WikiBigEdit with 500K edits, and ULTRAEDITBENCH with up to 2 million edits). They also tested general capabilities using the GLUE benchmark and domain generalization via the MedCF medical dataset.

The results highlight several major breakthroughs:

  • Massive Scalability: STABLEEDIT successfully executed 2 million sequential edits on Llama-3-8B. In contrast, baseline methods like MEMIT or MEND collapsed or became computationally infeasible long before reaching this scale.
  • Superior Stability: On the WikiBigEdit 500K task, STABLEEDIT improved generalization by 7.43% relative to the previous state-of-the-art (ULTRAEDIT) on GPT-J-6B.
  • Preserved General Intelligence: While non-LN methods saw their GLUE scores plummet to near-zero after just 30K edits, STABLEEDIT’s performance remained nearly identical to the original, unedited model.
  • Low Overhead: Despite the mathematical complexity of full whitening, STABLEEDIT is only 1.43x slower than the fastest known editor, making it vastly more efficient than hypernetwork-based alternatives that can be 20x to 80x slower.

Limitations and Future Directions

While the theoretical framework behind STABLEEDIT is robust, the authors note that it relies on the assumption that the gradient distribution evolves relatively smoothly. Abrupt, massive shifts in the underlying data distribution could still pose challenges for the running statistics.

Additionally, current LME evaluations rely heavily on token-matching metrics. Future research must explore more nuanced ways to measure whether an edit has genuinely updated the model's underlying belief system or merely altered its surface-level output. Finally, the authors suggest that hypernetwork-based editors (such as RLEDIT) require further theoretical exploration to fully align their "black box" mechanisms with the stability loops discovered in this work.

Summary updated

Abstract

Lifelong Model Editing aims to continuously update evolving facts in Large Language Models while preserving unrelated knowledge and general capabilities, yet it remains plagued by catastrophic forgetting and model collapse. Empirically, we find that recent editors resilient over long horizons share the same core strategy: **Lifelong Normalization (LN)**, which normalizes value gradients using running statistics. Removing LN causes immediate performance collapse, and we observe a counter-intuitive **positive cumulative effect** where early edits can promote the success of future edits. Yet the mechanism of LN remains a "black box", leaving its precise role in lifelong stability poorly understood. In this work, we provide the **first** theoretical account of LN in the lifelong regime. Our analysis reveals a self-reinforcing stability loop and proves that, when combined with ridge-regularized regression, LN yields parameter updates with **asymptotic orthogonality** and **bounded norms**, directly mitigating forgetting and systemic collapse. Based on these insights, we derive **StableEdit**, which strengthens this stability loop via an explicit warm-up stage and full whitening, improving long-horizon stability at minimal overhead. Extensive experiments validate our theory and demonstrate competitive performance. Our code is available at https://github.com/MINE-USTC/StableEdit.
BibTeX
@inproceedings{ma2026more,
  author = {Xin Ma and Wei Chen and Qi Liu and Derong Xu and Zhi Zheng and Tong Xu and Enhong Chen},
  title = {{More Edits, More Stable: Understanding the Lifelong Normalization in Sequential Model Editing}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=OcCpVIcgaY},
}
Figure