The Truth Beneath the Tokens: Prompt-Space Recovery of Knowledge Edits in Language Models

Manit Baser, Rachel Yeo Hui Min, Aditya Nawal, Dinil Mon Divakaran, Mohan GurusamyarXiv 2026 · arXiv

TL;DR

When we "edit" a fact in a Large Language Model (LLM), is the old knowledge truly deleted or just buried? This paper introduces MEMENTO, a technique that recovers "deleted" pre-edit knowledge using only specialized input prompts—without ever touching the model's weights. The results suggest that current model editing techniques primarily suppress factual accessibility rather than eliminating the underlying information.

What Problem Does This Paper Address?

As LLMs become central to our information infrastructure, we need ways to update them—correcting outdated facts or removing sensitive personal information. "Model editing" (like the popular ROME or MEMIT methods) offers a surgical alternative to expensive retraining by identifying and rewriting specific weights responsible for a fact.

However, a critical question remains: Does editing actually erase the old knowledge?

Previous attempts to "undo" edits relied on weight-space interventions—essentially hacking the model's parameters back to their original state. This paper investigates whether that knowledge is still lurking "beneath the tokens," recoverable through clever prompting alone. If it is, then model editing might be a less permanent "delete" than we previously thought, posing significant implications for privacy and model safety.

Methods

The authors propose MEMENTO, a prompt-space auditing mechanism. Instead of modifying the model $f_\theta$, they learn a reusable "recovery signal" $\Pi$ that is prepended to the input.

Figure 1: Overview of MEMENTO’s continuous prompt-space recovery. A trainable prefix Π is prepended to factual prompts and optimized across training facts to recover the original object o from an edited model fθ′, whose parameters remain frozen. The objective combines supervised recovery (Lrec), preference optimization (Lpref), fluency preservation (Lflu), and distillation from the pre-edit model (Ldist). Gradients update only Π; the pre-edit model supplies distillation targets during training and is not needed at inference. Without the prefix, the edited LLM answers the edited object (o∗= midfielder); with it, the LLM produces the original object (o = quarterback).

MEMENTO comes in two flavors:

  1. Continuous Recovery: Learns a "soft" prefix (a sequence of trainable embeddings). This is highly effective but requires access to the model's embedding layer.
  2. Discrete Recovery: Uses gradient-guided search (GCG) to find a sequence of actual text tokens that act as a "skeleton key" to unlock the pre-edit knowledge. This works even through standard text-only API interfaces.

To train these signals, the authors used a multi-part objective function:

  • Recovery Loss: Increases the likelihood of the original (pre-edit) answer.
  • Preference Loss: Specifically pushes the model to prefer the old answer over the new, edited one.
  • Fluency & Distillation: Ensures the prefix doesn't break the model's general language abilities or change its behavior on unrelated topics.

Datasets

The study utilizes the CounterFact dataset, a standard benchmark for model editing that contains thousands of factual triples (e.g., "The Eiffel Tower is in [Paris]").

  • Training: 1,000 facts to learn the recovery signal.
  • Evaluation: 1,000 held-out facts to test if the signal generalizes to unseen edits.
  • Models: Evaluated across GPT2-XL (1.5B), GPT-J (6B), and Llama3 (8B).
  • Editing Techniques: Tested against five different methods: MEMIT, AlphaEdit, PRUNE, RECT, and CoME.

Key Insights and Ideas

  • Suppression, Not Deletion: The most striking insight is that editing is often a "mask" rather than an "erasure." By simply changing the input context, MEMENTO can bypass the parameter updates and elicit the original response.
  • The Accessibility Gap: In a surprising twist, the authors applied MEMENTO to pristine, unedited models. They found it significantly boosted factual recall. This suggests that LLMs often "know" facts they fail to retrieve during standard inference—a phenomenon the authors call the "accessibility gap."
  • Cross-Method Generalization: A recovery signal trained on a model edited with MEMIT can often recover facts from a model edited with AlphaEdit. This implies that different editing techniques might be suppressing knowledge in similar, predictable ways.
  • Mechanistic Transparency: Using "logit lens" analysis, the authors showed that without MEMENTO, the original answer loses support in the later layers of the edited model. MEMENTO restores this support, allowing the original factual trajectory to re-emerge as the data flows through the network.

Key Results

  • High Recovery Rate: Continuous MEMENTO recovered 80.07% of held-out edited facts across all models and techniques, a 9.58% relative improvement over the previous state-of-the-art weight-masking baseline (One Mask).
  • Black-Box Auditing: The discrete (text-only) variant achieved up to 49.0% recovery, proving that "deleted" knowledge can be extracted through a standard chat interface.
  • Recall Boost: On unedited models, MEMENTO improved average factual recall by 53.9% relative to standard prompting.
Table 3: Top-1 recovery and CMR of MEMENTO and One Mask across three LLMs and five editing techniques.

As seen in the results table above, MEMENTO consistently outperforms the baseline (One Mask) in almost every category, particularly on more modern architectures like Llama3.

Limitations and Future Work

  • Internal Representation: While MEMENTO proves behavioral recoverability, it doesn't prove that the knowledge remains "intact" in its original internal representation. It just shows the model can still produce the answer.
  • Scope of Models: The study focused on models up to 8B parameters. Whether these findings scale to 70B+ or frontier models remains an open question.
  • Safety Implications: If "deleted" PII (Personally Identifiable Information) can be recovered via a prompt, current editing-based unlearning methods are insufficient for privacy compliance. Future work needs to focus on more robust "hard" deletion methods that are resistant to prompt-space recovery.

Summary updated

Abstract

Model editing provides an efficient way to update factual knowledge in transformer models. Recent approaches reverse edits by modifying, masking, or decomposing model weights, but make recovery dependent on weight-space interventions. We instead show that pre-edit knowledge can be recovered through input-space interventions alone, without modifying the edited model's weights. We introduce Memento, a prompt-space mechanism for auditing residual knowledge that learns a reusable recovery signal to steer an edited model toward its pre-edit response. Memento supports both continuous recovery prefixes and discrete textual prompts for standard text interfaces. Across three LLMs and five editing techniques, Memento recovers 80.07% of held-out edited facts, achieving a 9.58% relative improvement over baselines, suggesting that editing may suppress the facts without fully eliminating their recoverability. We further investigate whether prompt-space recovery extends beyond model editing. On pristine, unedited LLMs, Memento improves average factual recall by 53.9% relative to standard inference. These results demonstrate the potential of prompt-space recovery for auditing residual knowledge in edited models and investigating the gap between factual knowledge and its accessibility during inference. We will release our source code.
BibTeX
@misc{baser2026truth,
  author = {Manit Baser and Rachel Yeo Hui Min and Aditya Nawal and Dinil Mon Divakaran and Mohan Gurusamy},
  title = {{The Truth Beneath the Tokens: Prompt-Space Recovery of Knowledge Edits in Language Models}},
  year = {2026},
  eprint = {2611.10450},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.10450},
}
Figure