TL;DR
Model editing offers a surgical way to update knowledge in Large Language Models (LLMs) without expensive retraining, but how robust are these edits in the wild? This paper reveals a massive gap between benchmark success and practical reliability: edited LLMs easily revert to old knowledge or hallucinate when faced with multi-turn dialogues, rephrased prompts, or highly popular facts. Ultimately, the more deeply embedded a piece of knowledge is in the model's original training, the harder it is to edit robustly.
What Problem Does This Paper Address?
As the real world changes, LLMs quickly become outdated. Traditional retraining or fine-tuning is computationally prohibitive for daily updates, leading to the rise of "model editing" techniques (such as ROME, MEMIT, and SERAC) that directly modify specific factual associations within a model's parameters.
However, existing benchmarks evaluate these methods using short, direct, and static prompts. In real-world applications—like a conversational AI agent—users interact through complex, multi-turn dialogues, use pronouns, rephrase questions, or even express doubt. This paper investigates the robustness of model editing under these realistic conditions, exposing how easily edited knowledge can be disrupted.
Methods
The authors evaluate model editing robustness across three core dimensions:
- Dialogue Simulation (RQ1): To simulate a realistic conversational setting, they use GPT-4 to act as a "user" chatting with an edited LLM (Llama-2-7B-chat). The GPT-4 user subtly guides the conversation toward topics intersecting with the edited knowledge without explicitly stating the target answer, probing whether the model maintains its new memory or contradicts itself.
- Attacking Prompts (RQ2): They design adversarial prompt variations combining different contexts (Related context from Wikipedia, Noisy context, Simulated dialogue) and query formats (Direct prompts, Fill-in-the-blank cloze forms, and Reference resolution where subjects are replaced with pronouns). They also test prompts that explicitly raise doubts about the model's output.
- Intrinsic Feature Analysis (RQ3): They analyze how "knowledge popularity" affects editing success. Popularity is quantified using three metrics: Frequency (Wikipedia pageviews), Connection (number of edges in WikiData), and Co-occurrence (two-hop paths between subject and object).
Datasets
The evaluation utilizes three mainstream datasets:
- CounterFact: A dataset designed for evaluating counterfactual edits (e.g., changing the location of a landmark).
- zsRE (Zero-Shot Relation Extraction): A factual question-answering dataset used to evaluate standard knowledge updates.
- MQUAKE-T: A time-changing dataset used to simulate real-world temporal updates and assess multi-hop reasoning.
Key Insights and Ideas
- Knowledge Reversion, Not Erasure: Model editing doesn't actually erase the original memory; it merely suppresses it. During a multi-turn conversation, the model can easily trace back through intersecting knowledge webs and unearth the original, unedited fact.
- The Popularity Curse: Popular facts are deeply woven into the model's weight matrix. The more connected a piece of knowledge is, the more resistant it is to clean editing, leading to severe performance drops when prompts deviate from standard templates.
- Benchmark Optimism: Current editing benchmarks predominantly feature less popular knowledge. This creates an overly optimistic view of editing performance, severely underestimating the difficulty of editing facts that models have strongly memorized.
Key Results
- High Confusion and Hallucination: In dialogue simulations, 38% of successfully edited models reverted to their original knowledge mid-conversation, and 22% fell into self-contradiction. Furthermore, 78% of the sessions exhibited clear hallucinations triggered by the knowledge conflict.
- Vulnerability to Rephrasing: While locate-then-edit methods like ROME and MEMIT achieve near-perfect accuracy on direct prompts, their performance plummets under cloze queries, reference resolution, or user doubts.
- Impact of Popularity: As knowledge frequency and connection levels increase, editing performance on rephrased prompts drops significantly (causing accuracy gaps of up to 21% for ROME and 13% for MEMIT compared to averages).
- Mitigation Success: The authors experimented with simple fixes, finding that adding complex contexts during the editing sampling phase (+Samp.) or using a two-step disentanglement prompting method (+Disen.) significantly improved robustness, boosting ROME's average robustness score by up to 20.5%.
Limitations and Future Work
The authors acknowledge several limitations:
- Scope of Edits: The primary focus remains on single-instance edits rather than sequential mass editing, which is known to introduce even higher risks of knowledge distortion.
- Automated Evaluation: The dialogue evaluation relies on a GPT-4 simulated user; human attackers might find even more sophisticated ways to break edited models.
Future Directions: The paper calls for developing more robust editing algorithms that can handle complex multi-edit scenarios, preserve general LLM capabilities, and mitigate information security risks (such as knowledge reversion exposing confidential data).
Summary updated