TL;DR
Evaluating how well a Vision-Language Model (VLM) "forgets" private data is fundamentally broken: standard evaluation metrics systematically contradict one another, and multimodal architectures make this inconsistency worse. This paper exposes these metric contradictions, reveals a massive shared blind spot in how we measure knowledge erasure, and introduces the Unified Quality Score (UQS) to bring data-driven order to unlearning evaluation.
What Problem Does This Paper Address?
Under regulations like the GDPR, machine learning models must comply with the "right to be forgotten" through a process called machine unlearning. While unlearning has been heavily studied in text-only and classification models, evaluating it in multimodal Vision-Language Models (VLMs) is currently a wild west.
Researchers rely on various metrics—such as Forget Accuracy (FA), Retain Accuracy (RA), and Membership Inference Attacks (MIA)—to determine if a model has successfully forgotten target data while maintaining its general utility. However, these metrics often yield completely conflicting rankings of unlearning methods. A method praised as a top performer under one metric might be deemed the worst under another, making cross-paper comparisons unreliable and jeopardizing true regulatory compliance.
Methods
The authors conduct the first systematic study of metric reliability in multimodal unlearning by evaluating 36 unlearned configurations of the LLaVA-1.5-7B model across four prominent unlearning paradigms:
- Gradient Ascent (GA): Maximizes loss on the forget set.
- Random Labels (RL): Fine-tunes the model on corrupted/random answers.
- Fine-Tune Retain (FT): Retrains only on the remaining valid data.
- Saliency-based Unlearning (SalUn): Selectively updates influential parameters using a saliency mask.
To resolve the contradictions among metrics, they introduce the Unified Quality Score (UQS). UQS is a composite metric that aggregates individual scores using weights derived from each metric's Spearman correlation ($\rho$) with a gold-standard "oracle" model ($M^*$)—a counterfactual model retrained from scratch solely on the retained data.
Additionally, they formalize and operationalize Knowledge Recoverability (KR), a targeted probing framework designed to test whether forbidden knowledge is truly erased or merely superficially suppressed.
Datasets
The paper evaluates unlearning performance across three distinct Visual Question Answering (VQA) benchmarks:
- MLLMU-Bench: 500 entity-level VQA pairs.
- UnLOK-VQA: 505 knowledge-graph-grounded pairs.
- MMUBench: 300 samples covering 20 visual concepts.
- CIFAR-10: Used with a ResNet-18 model as a unimodal baseline to isolate and quantify the impact of multimodality.
Key Insights and Ideas
- The Metric Schism: Metrics split into two fundamentally opposing clusters: behavioral output metrics ({FA, RA, MIA}) and representation alignment metrics ({AD, JS}). Optimizing for one family actively harms the other. For example, Gradient Ascent ranks #1 on MIA but dead last (#4) on Activation Distance (AD).
- Multimodal Amplification: Metric disagreement is significantly worse in multimodal VQA (mean $\tau = 0.086$) than in unimodal classification (mean $\tau = 0.158$). The dual image-text pathways allow the visual encoder and language decoder to decouple, creating complex representational discrepancies unique to VLMs.
- The Illusion of Forgetting: Forget Accuracy (FA) is actually negatively reliable ($\rho = -0.418$). Methods that achieve a low (seemingly good) FA often do so via "output collapse" (catastrophic breakdown of response generation) rather than actual knowledge erasure, driving the model further away from the ideal oracle state.
- The Shared Blind Spot: All five standard metrics fail to detect if a model still holds forbidden knowledge internally. Through a pilot study using rephrased, indirect, or negation queries, the authors recovered "forgotten" information in 96% of cases where direct queries were successfully suppressed.
Key Results
- Systematic Contradiction: Kendall’s $\tau$ between Forget Accuracy (FA) and Activation Distance (AD) was $-0.26$, demonstrating a clear rank reversal. This contradiction was successfully replicated on a completely different architecture (BLIP-2 OPT-2.7B), proving it is structural rather than model-specific.
- Metric Reliability: When validated against the retrained oracle model, Retain Accuracy (RA) proved to be the most reliable metric ($\rho = 0.484, p = 0.003$). This results in UQS assigning the highest weight to RA ($w_{RA} = 0.656$) and MIA ($w_{MIA} = 0.304$), while heavily penalizing FA, AD, and JS ($w = 0.014$).
- UQS Stability: The proposed UQS successfully resolves the ranking contradictions, providing stable and robust method rankings ($\tau = 0.647 \pm 0.262$) even under 100 random Dirichlet weight perturbations. Under the unified score, Gradient Ascent ranks #1 overall.
Limitations and Future Work
- Architecture and Dataset Dependency: While the UQS framework itself is universal, the exact weights and correlations derived are model- and dataset-dependent. The full evaluation focused primarily on LLaVA-1.5-7B, meaning weights must be re-derived when the evaluation context changes.
- Scalability of Knowledge Recoverability (KR): Although the authors prove that KR is a critical missing dimension for true GDPR compliance, measuring it currently requires structured, per-sample manual probe generation. Developing automated, large-scale KR estimation remains an open and vital direction for future research.
Summary updated