On the Robustness of Editing Large Language Models

Xinbei Ma, Tianjie Ju, Jiyang Qiu, Zhuosheng Zhang, Hai Zhao, Lifeng Liu, Yulong Wang

TL;DR

Model editing offers a surgical way to update knowledge in Large Language Models (LLMs) without expensive retraining, but how robust are these edits in the wild? This paper reveals a massive gap between benchmark success and practical reliability: edited LLMs easily revert to old knowledge or hallucinate when faced with multi-turn dialogues, rephrased prompts, or highly popular facts. Ultimately, the more deeply embedded a piece of knowledge is in the model's original training, the harder it is to edit robustly.

What Problem Does This Paper Address?

As the real world changes, LLMs quickly become outdated. Traditional retraining or fine-tuning is computationally prohibitive for daily updates, leading to the rise of "model editing" techniques (such as ROME, MEMIT, and SERAC) that directly modify specific factual associations within a model's parameters.

However, existing benchmarks evaluate these methods using short, direct, and static prompts. In real-world applications—like a conversational AI agent—users interact through complex, multi-turn dialogues, use pronouns, rephrase questions, or even express doubt. This paper investigates the robustness of model editing under these realistic conditions, exposing how easily edited knowledge can be disrupted.

Figure 1: Overview of our work. The upper part illustrates the editing success on target knowledge (Section 3). The lower part shows our studies on the edited model in realistic use. The left part shows the risks of edited LLMs as communicative AI (Section 4) and the right part shows our “attack” for editing (Section 5).

Methods

The authors evaluate model editing robustness across three core dimensions:

  1. Dialogue Simulation (RQ1): To simulate a realistic conversational setting, they use GPT-4 to act as a "user" chatting with an edited LLM (Llama-2-7B-chat). The GPT-4 user subtly guides the conversation toward topics intersecting with the edited knowledge without explicitly stating the target answer, probing whether the model maintains its new memory or contradicts itself.
  2. Attacking Prompts (RQ2): They design adversarial prompt variations combining different contexts (Related context from Wikipedia, Noisy context, Simulated dialogue) and query formats (Direct prompts, Fill-in-the-blank cloze forms, and Reference resolution where subjects are replaced with pronouns). They also test prompts that explicitly raise doubts about the model's output.
  3. Intrinsic Feature Analysis (RQ3): They analyze how "knowledge popularity" affects editing success. Popularity is quantified using three metrics: Frequency (Wikipedia pageviews), Connection (number of edges in WikiData), and Co-occurrence (two-hop paths between subject and object).

Datasets

The evaluation utilizes three mainstream datasets:

  • CounterFact: A dataset designed for evaluating counterfactual edits (e.g., changing the location of a landmark).
  • zsRE (Zero-Shot Relation Extraction): A factual question-answering dataset used to evaluate standard knowledge updates.
  • MQUAKE-T: A time-changing dataset used to simulate real-world temporal updates and assess multi-hop reasoning.

Key Insights and Ideas

  • Knowledge Reversion, Not Erasure: Model editing doesn't actually erase the original memory; it merely suppresses it. During a multi-turn conversation, the model can easily trace back through intersecting knowledge webs and unearth the original, unedited fact.
  • The Popularity Curse: Popular facts are deeply woven into the model's weight matrix. The more connected a piece of knowledge is, the more resistant it is to clean editing, leading to severe performance drops when prompts deviate from standard templates.
  • Benchmark Optimism: Current editing benchmarks predominantly feature less popular knowledge. This creates an overly optimistic view of editing performance, severely underestimating the difficulty of editing facts that models have strongly memorized.

Key Results

  • High Confusion and Hallucination: In dialogue simulations, 38% of successfully edited models reverted to their original knowledge mid-conversation, and 22% fell into self-contradiction. Furthermore, 78% of the sessions exhibited clear hallucinations triggered by the knowledge conflict.
Figure 2: Edited communicative AI. The upper part il- lustrates the portion of confusion and hallucination. The bottom shows a case that appears knowledge reversion.
  • Vulnerability to Rephrasing: While locate-then-edit methods like ROME and MEMIT achieve near-perfect accuracy on direct prompts, their performance plummets under cloze queries, reference resolution, or user doubts.
  • Impact of Popularity: As knowledge frequency and connection levels increase, editing performance on rephrased prompts drops significantly (causing accuracy gaps of up to 21% for ROME and 13% for MEMIT compared to averages).
Figure 4: Editing performance on different levels of (a) Frequency, (b) Connection, and (c) Co-occurrence.
  • Mitigation Success: The authors experimented with simple fixes, finding that adding complex contexts during the editing sampling phase (+Samp.) or using a two-step disentanglement prompting method (+Disen.) significantly improved robustness, boosting ROME's average robustness score by up to 20.5%.

Limitations and Future Work

The authors acknowledge several limitations:

  • Scope of Edits: The primary focus remains on single-instance edits rather than sequential mass editing, which is known to introduce even higher risks of knowledge distortion.
  • Automated Evaluation: The dialogue evaluation relies on a GPT-4 simulated user; human attackers might find even more sophisticated ways to break edited models.

Future Directions: The paper calls for developing more robust editing algorithms that can handle complex multi-edit scenarios, preserve general LLM capabilities, and mitigate information security risks (such as knowledge reversion exposing confidential data).

Summary updated

Abstract

Large language models (LLMs) have played a pivotal role in building communicative AI, yet they encounter the challenge of efficient updates. Model editing enables the manipulation of specific knowledge memories and the behavior of language generation without retraining. However, the robustness of model editing remains an open question. This work seeks to understand the strengths and limitations of editing methods, facilitating practical applications of communicative AI. We focus on three key research questions. RQ1: Can edited LLMs behave consistently resembling communicative AI in realistic situations? RQ2: To what extent does the rephrasing of prompts lead LLMs to deviate from the edited knowledge memory? RQ3: Which knowledge features are correlated with the performance and robustness of editing? Our empirical studies uncover a substantial disparity between existing editing methods and the practical application of LLMs. On rephrased prompts that are flexible but common in realistic applications, the performance of editing experiences a significant decline. Further analysis shows that more popular knowledge is memorized better, easier to recall, and more challenging to edit effectively. Code is publicly available at https://github.com/xbmxb/edit_analysis .
BibTeX
@inproceedings{ma2024robustness,
  author = {Xinbei Ma and Tianjie Ju and Jiyang Qiu and Zhuosheng Zhang and Hai Zhao and Lifeng Liu and Yulong Wang},
  title = {{On the Robustness of Editing Large Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.906},
}
Figure