Why Surface Metrics Fail to Verify Machine Unlearning

32 concepts23 papers1 proposal

Machine unlearning aims to act as a digital eraser, surgically removing sensitive data, copyrighted text, or harmful behaviors from a trained model's memory. To evaluate whether an unlearning algorithm has succeeded, researchers typically look at behavioral outputs: Does an LLM still output the forbidden text? Does a classifier's accuracy on the "forget set" drop to zero?

If the model stops producing the target data, standard evaluation metrics register a success. However, a growing body of research demonstrates that this output-centric view creates a dangerous illusion. Rather than erasing knowledge, many unlearning algorithms merely implement a "gag order" or induce "fake forgetting." The underlying information remains fully intact within the model's internal representations, waiting to be extracted.

The Mechanics of the Gag Order

Why do models appear to forget while retaining knowledge? The core reason lies in how deep networks process information. Unlearning often modifies only the final layers or retrieval mechanisms, leaving the core feature extractors untouched.

As Yihuai Hong et al. show in Dissecting Fine-Tuning Unlearning in Large Language Models, fine-tuning-based unlearning in Large Language Models (LLMs) rarely erases memory. Instead, it simply alters MLP coefficients in the final layers. This breaks the retrieval pathway—effectively issuing a gag order—but leaves the internal representations intact and accessible through alternative means. Similarly, Seonguk Seo et al. demonstrate in Revisiting Machine Unlearning with Dimensional Alignment that a model can easily manipulate its final predictions to hide sensitive data characteristics while preserving highly discriminative representations in its internal layers.

Because these internal features remain intact, they can be easily exploited. Tae-Young Lee et al. find in ESC: Erasing Space Concept for Knowledge Deletion that even when a model fails at classification, simple linear probes can recover the supposedly deleted knowledge from its internal features. This vulnerability extends to transfer learning: Seunghee Koh et al. note in Disposable Transfer Learning for Selective Source Task Unlearning that a model can fail at a source task while still harboring useful internal features that allow unauthorized "piggyback" learning on downstream tasks.

[Input Prompt] ──> [Internal Layers: Features Intact] ──> [Final Layers: "Gag Order"] ──> [Output: "I don't know"]
                                 │
                                 └───(Adversarial Probe / Jailbreak)───> [Restored Knowledge]

How Models Game Surface-Level Metrics

When evaluated using standard string-matching or token-level metrics like ROUGE or perplexity, models can appear to have forgotten data through simple optimization shortcuts rather than genuine conceptual removal.

One prominent shortcut is the squeezing effect. As Kemou Li et al. explain in LLM Unlearning with LLM Beliefs, suppressing a specific target response often just pushes the model's probability mass into semantically identical rephrasings. Standard metrics like ROUGE measure surface-level string overlap and are completely blind to this shift, falsely reporting successful forgetting when the model can still generate the restricted information using alternative phrasing.

Alternatively, models can game metrics by degrading their output quality entirely. Xinrui Chen et al. discover in Robust LLM Unlearning via Post Judgment and Multi-round Thinking that models can artificially inflate unlearning scores by generating unreadable or nonsensical text instead of demonstrating a true removal of knowledge. Similarly, in generative vision models, Christoforos N. Spartalis et al. point out in Unleashing Uncertainty: Efficient Machine Unlearning for Generative AI that unlearning accuracy metrics can be highly misleading, as downstream classifiers might misidentify pure Gaussian noise as a specific class, masking whether the underlying concept was truly purged.

The Blind Spots of Privacy Audits and Perplexity

Even more sophisticated evaluation techniques, such as Membership Inference Attacks (MIAs) and perplexity tracking, suffer from systemic blind spots.

Perplexity Insensitivity

Chonghua Liao et al. reveal in Exploring Forgetting in Large Language Model Pre-Training that perplexity is heavily influenced by high-frequency, predictable tokens. This makes it insensitive to the loss or retention of fine-grained, specific factual knowledge, hiding critical memory dynamics and creating a false impression of continuous improvement.

MIA Inadequacy

MIAs are designed to check if a specific sample was part of a training set. However, Martin Pawelczyk et al. demonstrate in Machine Unlearning Fails to Remove Data Poisoning Attacks that a model can pass an MIA audit perfectly while still retaining the full functional and adversarial impact of a data poisoning attack. Dasol Choi et al. echo this in Distribution-Level Feature Distancing for Machine Unlearning: Towards a Better Trade-off Between Model Utility and Forgetting, arguing that MIAs fail to comprehensively capture the true extent of data removal across diverse scenarios beyond simple membership detection.

Hidden Confidence Sets

Yingdan Shi et al. identify a phenomenon called "fake forgetting" in Tackling Fake Forgetting through Uncertainty Quantification. They show that while a model might misclassify an item on the surface (satisfying accuracy-based metrics), it often still retains the true label within its high-confidence prediction sets.

Metric Type What It Measures Why It Fails
Accuracy / ROUGE Surface-level string or label match Blind to semantic rephrasings, gibberish generation, and internal feature retention.
Perplexity Average token predictability Dominated by high-frequency tokens; insensitive to fine-grained factual decay.
Membership Inference (MIA) Data sample presence Measures membership rather than whether the functional impact (e.g., poisoning) is neutralized.

Collateral Damage: Knowledge Holes and Over-Unlearning

Because current metrics focus narrowly on whether a specific target is suppressed, they fail to measure the broader systemic damage caused by unlearning algorithms. Unlearning is rarely surgical; it often creates massive, unintended "knowledge holes" in the model's remaining capabilities.

Myeongseob Ko et al. show in Probing Hidden Knowledge Holes in Unlearned LLMs that aggregate benchmark scores can remain completely stable even when a model loses nearly all functional utility in specific semantic neighborhoods. Furthermore, Weixiang Zhao et al. note in The gains do not make up for the losses: a comprehensive evaluation for safety alignment of large language models via machine unlearning that narrow evaluations overlook how unlearning can leave models incoherent or overly defensive.

This problem is compounded when datasets overlap: Xinyang Lu et al. point out in WaterDrum: Watermark-based Data-centric Unlearning Metric that traditional utility metrics cannot distinguish between forgotten data and retained data when they share high semantic similarity, leading to highly inaccurate safety assessments.

Moving Toward Robust Verification

To move past the illusion of surface-level forgetting, researchers are developing metrics that probe a model's internal state and adversarial resilience:

Key Foundational Work

An Adversarial Perspective on Machine Unlearning for AI Safety (2024) — contributed 3 concept elements to this direction.
Dissecting Fine-Tuning Unlearning in Large Language Models (2024) — contributed 1 concept elements to this direction.
Jogging the Memory of Unlearned Models Through Targeted Relearning Attacks (2024) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_advanced_privacy_attack_evaluation_d5f97fc6,
  title = {advanced privacy attack evaluation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/advanced-privacy-attack-evaluation/rdr_d5f97fc6bebb4ecaa2b77469f6fca9b6},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_advanced_privacy_attack_evaluation_d5f97fc6_v1,
  title = {advanced privacy attack evaluation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/advanced-privacy-attack-evaluation/rdr_d5f97fc6bebb4ecaa2b77469f6fca9b6/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake