CoME: An Unlearning-based Approach to Conflict-free Model Editing

Dahyun Jung, Jaehyung Seo, Jaewook Lee, Chanjun Park, Heuiseok Lim

Abstract

Large language models (LLMs) often retain outdated or incorrect information from pre-training, which undermines their reliability. While model editing methods have been developed to address such errors without full re-training, they frequently suffer from knowledge conflicts, where outdated information interferes with new knowledge. In this work, we propose Conflict-free Model Editing (CoME), a novel framework that enhances the accuracy of knowledge updates in LLMs by selectively removing outdated knowledge. CoME leverages unlearning to mitigate knowledge interference, allowing new information to be integrated without compromising relevant linguistic features. Through experiments on GPT-J and LLaMA-3 using Counterfact and ZsRE datasets, we demonstrate that CoME improves both editing accuracy and model reliability when applied to existing editing methods. Our results highlight that the targeted removal of outdated knowledge is crucial for enhancing model editing effectiveness and maintaining the model's generative performance.

Introduction: The Challenge of Knowledge Conflict

Updating an LLM's knowledge without retraining the entire model is a critical task known as Model Editing. While state-of-the-art methods like MEMIT and PMET are effective at "locating and editing" specific facts, they often leave behind "ghosts" of the old information. This leads to knowledge interference, where the model experiences conflict between pre-trained outdated facts and newly injected information. For example, if a model originally learned "The language of El Correo is Spanish" and is edited to "English," it might produce garbled responses like "English Spanish" because internal parameters for both facts compete.

Conceptual comparison of standard model editing versus CoME's unlearning approach.

Methodology: Conflict-free Model Editing (CoME)

The core innovation of CoME is a parameter subtraction-based unlearning framework. Instead of merely adding new information, CoME actively subtracts the old knowledge to clear the path for the new. The process involves three key components:

  1. Extraction of Outdated Parameters: The framework calculates a "negative update" vector ($\delta'_i$), representing the parameter adjustments associated with the outdated fact.
  2. Linguistic Feature Preservation: To avoid degrading the model's general language capabilities, CoME uses vector projection. It identifies the "common component" between old and new knowledge vectors, assuming this shared space represents general linguistic ability, and ensures this part is preserved during subtraction.
  3. Restricted Unlearning: To maintain Locality (preventing collateral damage to unrelated facts), CoME restricts the unlearning process to the top-p% (typically 20%) of parameters most influenced by the outdated knowledge.

By performing subtraction and addition simultaneously, the model transitions directly to the new state, avoiding the instability often found in sequential "forget then learn" approaches.

Experimental Evaluation

The authors evaluated CoME using two standard benchmarks:

  • Counterfact: 10,000 samples of counterfactual statements (e.g., "The Eiffel Tower is in Rome") to test overriding pre-trained "truths."
  • ZsRE (Zero-Shot Relation Extraction): 10,000 samples evaluating factual relation edits in context-free settings.

Experiments were conducted on GPT-J (6B) and LLaMA-3 (8B), integrating CoME with existing editors like MEMIT and PMET.

Key Results and Discussion

CoME consistently improved performance across multiple metrics, including Score, Efficacy, and Generality.

Performance comparison of CoME against baseline methods on the Counterfact dataset.
  • Quantitative Gains: On the Counterfact dataset with LLaMA-3, CoME combined with PMET achieved a Score of 82.3, outperforming the standard PMET baseline (81.1).
  • Enhanced Generality: In ZsRE tests on LLaMA-3, CoME-PMET reached an Efficacy of 90.6%, showing significant improvements in the model's ability to recognize updated facts across different phrasings.
  • Qualitative Improvements: Case studies revealed that while baseline methods sometimes produced conflicting outputs (e.g., "English Spanish"), CoME-edited models generated clean, contextually relevant sentences about the new fact.

Conclusion and Future Work

CoME demonstrates that the active weakening of conflicting memories—a process mirrored in human neurobiology—is essential for effective model editing. While CoME introduces some computational overhead due to the outdated vector calculation, its ability to provide "clean" updates makes it a significant advancement. Future work includes exploring the framework's performance on larger models (70B+) and refining the balance between correcting facts and preserving historical knowledge where relevant.

Summary updated

Abstract

Large language models (LLMs) often retain outdated or incorrect information from pre-training, which undermines their reliability. While model editing methods have been developed to address such errors without full re-training, they frequently suffer from knowledge conflicts, where outdated information interferes with new knowledge. In this work, we propose Conflict-free Model Editing (CoME), a novel framework that enhances the accuracy of knowledge updates in LLMs by selectively removing outdated knowledge. CoME leverages unlearning to mitigate knowledge interference, allowing new information to be integrated without compromising relevant linguistic features. Through experiments on GPT-J and LLaMA-3 using Counterfact and ZsRE datasets, we demonstrate that CoME improves both editing accuracy and model reliability when applied to existing editing methods. Our results highlight that the targeted removal of outdated knowledge is crucial for enhancing model editing effectiveness and maintaining the model's generative performance.
BibTeX
@inproceedings{jung2025come,
  author = {Dahyun Jung and Jaehyung Seo and Jaewook Lee and Chanjun Park and Heuiseok Lim},
  title = {{CoME: An Unlearning-based Approach to Conflict-free Model Editing}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
  url = {https://openreview.net/forum?id=FEfAJzWVw7},
}
Figure