Metric Unreliability in Multimodal Machine Unlearning: A Systematic Analysis and Principled Unified Score

Abdullah Ahmad Khan, Hamid Laga, Ferdous SohelarXiv 2026 · arXiv

TL;DR

Evaluating how well a Vision-Language Model (VLM) "forgets" private data is fundamentally broken: standard evaluation metrics systematically contradict one another, and multimodal architectures make this inconsistency worse. This paper exposes these metric contradictions, reveals a massive shared blind spot in how we measure knowledge erasure, and introduces the Unified Quality Score (UQS) to bring data-driven order to unlearning evaluation.

What Problem Does This Paper Address?

Under regulations like the GDPR, machine learning models must comply with the "right to be forgotten" through a process called machine unlearning. While unlearning has been heavily studied in text-only and classification models, evaluating it in multimodal Vision-Language Models (VLMs) is currently a wild west.

Researchers rely on various metrics—such as Forget Accuracy (FA), Retain Accuracy (RA), and Membership Inference Attacks (MIA)—to determine if a model has successfully forgotten target data while maintaining its general utility. However, these metrics often yield completely conflicting rankings of unlearning methods. A method praised as a top performer under one metric might be deemed the worst under another, making cross-paper comparisons unreliable and jeopardizing true regulatory compliance.

Methods

The authors conduct the first systematic study of metric reliability in multimodal unlearning by evaluating 36 unlearned configurations of the LLaVA-1.5-7B model across four prominent unlearning paradigms:

  • Gradient Ascent (GA): Maximizes loss on the forget set.
  • Random Labels (RL): Fine-tunes the model on corrupted/random answers.
  • Fine-Tune Retain (FT): Retrains only on the remaining valid data.
  • Saliency-based Unlearning (SalUn): Selectively updates influential parameters using a saliency mask.

To resolve the contradictions among metrics, they introduce the Unified Quality Score (UQS). UQS is a composite metric that aggregates individual scores using weights derived from each metric's Spearman correlation ($\rho$) with a gold-standard "oracle" model ($M^*$)—a counterfactual model retrained from scratch solely on the retained data.

Additionally, they formalize and operationalize Knowledge Recoverability (KR), a targeted probing framework designed to test whether forbidden knowledge is truly erased or merely superficially suppressed.

Datasets

The paper evaluates unlearning performance across three distinct Visual Question Answering (VQA) benchmarks:

  • MLLMU-Bench: 500 entity-level VQA pairs.
  • UnLOK-VQA: 505 knowledge-graph-grounded pairs.
  • MMUBench: 300 samples covering 20 visual concepts.
  • CIFAR-10: Used with a ResNet-18 model as a unimodal baseline to isolate and quantify the impact of multimodality.

Key Insights and Ideas

  • The Metric Schism: Metrics split into two fundamentally opposing clusters: behavioral output metrics ({FA, RA, MIA}) and representation alignment metrics ({AD, JS}). Optimizing for one family actively harms the other. For example, Gradient Ascent ranks #1 on MIA but dead last (#4) on Activation Distance (AD).
  • Multimodal Amplification: Metric disagreement is significantly worse in multimodal VQA (mean $\tau = 0.086$) than in unimodal classification (mean $\tau = 0.158$). The dual image-text pathways allow the visual encoder and language decoder to decouple, creating complex representational discrepancies unique to VLMs.
  • The Illusion of Forgetting: Forget Accuracy (FA) is actually negatively reliable ($\rho = -0.418$). Methods that achieve a low (seemingly good) FA often do so via "output collapse" (catastrophic breakdown of response generation) rather than actual knowledge erasure, driving the model further away from the ideal oracle state.
  • The Shared Blind Spot: All five standard metrics fail to detect if a model still holds forbidden knowledge internally. Through a pilot study using rephrased, indirect, or negation queries, the authors recovered "forgotten" information in 96% of cases where direct queries were successfully suppressed.

Key Results

  • Systematic Contradiction: Kendall’s $\tau$ between Forget Accuracy (FA) and Activation Distance (AD) was $-0.26$, demonstrating a clear rank reversal. This contradiction was successfully replicated on a completely different architecture (BLIP-2 OPT-2.7B), proving it is structural rather than model-specific.
  • Metric Reliability: When validated against the retrained oracle model, Retain Accuracy (RA) proved to be the most reliable metric ($\rho = 0.484, p = 0.003$). This results in UQS assigning the highest weight to RA ($w_{RA} = 0.656$) and MIA ($w_{MIA} = 0.304$), while heavily penalizing FA, AD, and JS ($w = 0.014$).
  • UQS Stability: The proposed UQS successfully resolves the ranking contradictions, providing stable and robust method rankings ($\tau = 0.647 \pm 0.262$) even under 100 random Dirichlet weight perturbations. Under the unified score, Gradient Ascent ranks #1 overall.

Limitations and Future Work

  • Architecture and Dataset Dependency: While the UQS framework itself is universal, the exact weights and correlations derived are model- and dataset-dependent. The full evaluation focused primarily on LLaVA-1.5-7B, meaning weights must be re-derived when the evaluation context changes.
  • Scalability of Knowledge Recoverability (KR): Although the authors prove that KR is a critical missing dimension for true GDPR compliance, measuring it currently requires structured, per-sample manual probe generation. Developing automated, large-scale KR estimation remains an open and vital direction for future research.

Summary updated

Abstract

Machine unlearning in Vision-Language Models (VLMs) is required for compliance with the General Data Protection Regulation (GDPR), yet current evaluation practices are inconsistent. We present the first systematic study of metric reliability in multimodal unlearning. Five standard metrics, Forget Accuracy (FA), Retain Accuracy (RA), Membership Inference Attack (MIA), Activation Distance (AD), and JS divergence (JS), yield conflicting method rankings across three VQA benchmarks (MLLMU-Bench, UnLOK-VQA, MMUBench). Kendall tau analysis over 36 unlearned LLaVA-1.5-7B models reveals two opposing clusters, {FA, RA, MIA} and {AD, JS}, with tau_FA_AD = -0.26, reproduced on BLIP-2 OPT-2.7B. Agreement is lower in multimodal VQA (average tau = 0.086) than in unimodal classification (average tau = 0.158; difference = 0.072), indicating that dual image-and-text pathways amplify inconsistency. We introduce the Unified Quality Score (UQS), a composite metric with weights derived from each metric's Spearman correlation with the oracle distance d(M_hat, M_star), where M_star is the oracle model retrained only on the retain set. RA shows the strongest reliability (rho = 0.484, p = 0.003), while FA is negatively correlated (rho = -0.418, p = 0.011). UQS yields stable rankings under 100 random weight perturbations (tau = 0.647 +- 0.262). We release the benchmark, 36 checkpoints, and an interactive leaderboard. Code and pre-computed results are available at https://github.com/neurips26/UnifiedUnl.
BibTeX
@misc{khan2026metric,
  author = {Abdullah Ahmad Khan and Hamid Laga and Ferdous Sohel},
  title = {{Metric Unreliability in Multimodal Machine Unlearning: A Systematic Analysis and Principled Unified Score}},
  year = {2026},
  eprint = {2605.02206},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.02206},
}
Figure