MuEdit: An efficient multi-task editing method towards inter-domain knowledge conflicts

Songshi Liang, Ting-En Lin, Zixuan Huang, Yuchuan Wu, Yongbin Li, Zihe Wang, Rui YanarXiv 2026 · arXiv

This paper, presented at NeurIPS 2026, introduces Mu-Edit, a novel framework designed to solve a critical problem in Large Language Model (LLM) maintenance: how to update a model across multiple, diverse domains (like factual knowledge, safety, and reasoning) without the model "forgetting" previous updates or losing its general capabilities.

TL;DR

Mu-Edit addresses the "multi-task editing collapse" in LLMs by identifying and quantifying geometric conflicts between different task domains using a new Conflict Index. By combining conflict-aware task ordering with adaptive low-rank approximations of preservation constraints, Mu-Edit enables stable, sequential updates across heterogeneous domains while maintaining superior general model performance compared to existing state-of-the-art methods.

What Problem Does This Paper Address?

LLMs are not static; they require frequent updates to correct facts, improve safety, or add new capabilities. Model editing aims to make these changes efficiently without full retraining. However, most existing methods are optimized for sequential editing within a single domain (e.g., correcting 1,000 facts).

When these methods are applied across heterogeneous domains (e.g., first editing for safety, then for math, then for sentiment), they often suffer from catastrophic interference. The authors trace this failure to inter-domain null-space misalignment.

In technical terms, each task requires the model to preserve certain internal representations. These requirements act as geometric constraints, defining a "null space" where updates are allowed to happen without breaking old knowledge. As you add more diverse tasks, the intersection of these null spaces—the "common null space"—shrinks rapidly. Eventually, there are no "safe" directions left to fit new information, leading to a performance collapse.

Figure 1: Mechanism and motivation of our Mu-Edit, specifically: (a) Visualization of Mu-Edit’s null space projection mechanism. (b) Empirical validation results of existing methods’ performance decline after editing sequentially across different tasks.

Methods

Mu-Edit introduces a two-pronged strategy to manage this geometric bottleneck:

  1. The Conflict Index (CI): The authors propose a metric to quantify the geometric overlap between task-specific null spaces. It measures the relative loss of admissible update directions when two tasks are imposed jointly.
  2. Conflict-Aware Ordering:
    • Best Order (Offline): If all tasks are known in advance, Mu-Edit searches for a sequence that minimizes the sum of adjacent pairwise conflicts.
  3. Greedy Order (Online): For incremental updates, a new task is inserted into the existing sequence at the position that minimizes local conflict, using a "local protection window" (typically the 3 most recent tasks) to balance preservation and flexibility.
  4. Dynamic Low-Rank Approximation: When ordering isn't enough to resolve a high-conflict pair, Mu-Edit adaptively compresses the preservation matrix of the more "burdensome" task. By discarding less significant singular values, it relaxes the constraints and expands the available null space for future updates.
  5. Closed-Form Updates: The final parameter update is computed using a Moore-Penrose pseudo-inverse formulation that fits new associations while regularizing against changes to the representations of tasks within the local protection window.

Datasets

The authors evaluated Mu-Edit across five primary functional domains using three different model backbones (Llama3-8B, Qwen2.5-7B, and GPT2-xl):

  • Common-sense/Factual: ZsRE, CounterFact, and CKnowEdit (Chinese).
  • Safety: SafeEdit.
  • Reasoning: SQuAD and GSM8K.
  • Sentiment: SST-2.
  • General Ability: Evaluated using SAMSum (summarization) and Natural Questions (QA).
  • Extended Testing: A seven-task sequence including Personality and LongCodeQA.

Key Insights and Ideas

  • Task Order is a First-Class Citizen: The sequence in which you edit a model significantly impacts final performance. Some task transitions are "mild," while others are "catastrophic."
  • The "Common Null Space" Bottleneck: The primary reason multi-task editing fails isn't just the number of edits, but the heterogeneity of the constraints. Diverse tasks "fight" for the same parameter dimensions.
  • Greedy is Good Enough: While finding the absolute "Best Order" is computationally expensive ($N!$ permutations), the Greedy Order strategy achieves nearly identical performance with much lower overhead, making it viable for real-world deployment.
  • Strategic Relaxation: You don't need to preserve every internal representation perfectly. Moderate rank reduction (up to 30-45%) in the preservation matrices can significantly ease conflicts without noticeably harming the model's performance on the original tasks.

Key Results

Mu-Edit consistently outperformed baselines like MEMIT, PRUNE, and AlphaEdit across all metrics.

  • Superior Multi-Task Stability: On Llama3-8B, Mu-Edit achieved a ZsRE Specificity of 0.8845, compared to AlphaEdit's 0.5506 and AnyEdit's 0.7098 under the same "Best Order" conditions.
  • General Ability Retention: While methods like MEMIT saw general capability scores drop to near zero after just two tasks, Mu-Edit showed only minor declines, proving it doesn't "break" the base model's core logic.
  • Efficiency: The Greedy Order approach reduced ordering time for 10 tasks from 5,502 minutes (Best Order) to just 22.9 seconds, while maintaining performance within 1.4% of the offline optimum.
Table 1: Final performance after multi-task editing on Llama3-8B under Best Order(BO) and Default Order(DO) and ablations. Bold: best. *: significant improvement over AnyEdit (p < 0.05).

Limitations and Future Work

  • Hyperparameter Sensitivity: The method relies on several tuned parameters, including the conflict threshold ($\mu$) and the adaptive thresholding coefficient ($t$). The authors suggest that future work could focus on a fully parameter-free, self-adaptive scheduler.
  • Extreme Conflicts: In cases of "catastrophic geometric misalignment" where all tasks are highly conflicting, the dynamic rank reduction might eventually destabilize the model if forced to compress too aggressively.
  • Layer Selection: The current work targets specific MLP layers (e.g., layers 4–8 in Llama3). A more automated way to select optimal layers for different task types remains an open question.

Summary updated

Abstract

Large language models (LLMs) require frequent knowledge updates across heterogeneous domains, but existing editing methods fail when applied sequentially to multi-task settings. We trace this collapse to inter-domain null-space misalignment: heterogeneous tasks induce distinct Jacobian geometries, causing their common null space to shrink dramatically and leaving insufficient admissible directions for conflict-free updates. We propose the Conflict Index to quantify this geometric interference, and introduce Mu-Edit, which mitigates multi-task conflicts via (1) conflict-aware ordering to minimize cumulative interference, and (2) dynamic low-rank approximation to expand the null space when ordering alone is insufficient. Experiments across five functional domains and three backbones show that Mu-Edit significantly outperforms existing baselines in multi-task editing performance while preserving general capabilities, and remains effective under incremental task arrival.
BibTeX
@misc{liang2026muedit,
  author = {Songshi Liang and Ting-En Lin and Zixuan Huang and Yuchuan Wu and Yongbin Li and Zihe Wang and Rui Yan},
  title = {{MuEdit: An efficient multi-task editing method towards inter-domain knowledge conflicts}},
  year = {2026},
  eprint = {2611.14910},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.14910},
}
Figure