Unlearning Comparator: A Visual Analytics System for Comparative Evaluation of Machine Unlearning Methods

Jaeung Lee, Suhyeon Yu, Yurim Jang, Simon S. Woo, Jaemin Jo

Introduction

As data privacy regulations like the GDPR establish the "right to be forgotten," AI developers face the challenge of Machine Unlearning (MU)—the process of removing specific training data from a model's weights. While retraining from scratch is the gold standard for privacy, it is often computationally prohibitive. Consequently, researchers rely on "approximate unlearning" methods. However, verifying that a model has truly forgotten data without degrading its overall performance remains a "black box" problem. Standard aggregate metrics often fail to detect residual signals that leave models vulnerable to Membership Inference Attacks (MIAs).

To address this, we present Unlearning Comparator, a visual analytics system designed to facilitate granular, multi-level comparisons between unlearning methods. This tool enables researchers to move beyond surface-level accuracy and understand the internal representational changes occurring during the unlearning process.

System Design and Workflow

The Unlearning Comparator is built around a four-stage iterative workflow that guides users from model generation to rigorous privacy testing.

The four-stage workflow of the Unlearning Comparator: Build, Screen, Contrast, and Attack.
  1. Build: Users generate candidate models by varying hyperparameters such as learning rate and batch size across different unlearning algorithms (e.g., Fine-Tuning, Gradient Ascent).
  2. Screen: A high-level dashboard allows users to filter models based on three primary pillars: accuracy, efficiency, and privacy.
  3. Contrast: This core component provides side-by-side comparisons at the class level (accuracy trade-offs), layer level (internal representation similarity), and instance level (data point distribution via UMAP).
  4. Attack: The system simulates MIAs to calculate the Worst-Case Privacy Score (WCPS), a conservative metric that aggregates confidence and entropy-based attacks to identify vulnerabilities that standard metrics miss.

Key Insights: The "Elbow Layer"

By utilizing the system's Layer-wise Similarity view, which employs Centered Kernel Alignment (CKA), researchers identified a critical phenomenon: the "Elbow Layer." This is a transition point in the neural network where the model shifts from learning general features to specialized, class-specific representations.

Layer-wise similarity analysis using CKA values to identify the "elbow layer" where internal representations diverge.

Analysis revealed that re-initializing layers after this elbow is significantly more efficient than adjusting the entire network. Furthermore, the tool highlighted "confidence mismatches" in common methods like Random Labeling, where models might predict a class correctly but with dangerously low confidence—a subtle indicator of incomplete unlearning.

Guided Unlearning (GU) and Results

Insights gained from the Unlearning Comparator led to the development of Guided Unlearning (GU). GU is a hybrid approach that combines targeted weight re-initialization at the elbow layer, tuned gradient ascent, and a "recovery" fine-tuning stage to stabilize the model.

Evaluated on CIFAR-10 and Fashion-MNIST using ResNet-18 and ViT architectures, GU demonstrated superior performance:

  • Perfect Forgetting: Achieved 0.000 Unlearning Accuracy while maintaining 0.994 Retain Accuracy.
  • Enhanced Privacy: Reached a WCPS of 0.806, outperforming the standard Gradient Ascent + Fine-Tuning baseline (0.663).
  • Efficiency: The targeted layer strategy resulted in 30% faster convergence than standard fine-tuning.

Conclusion and Future Work

The Unlearning Comparator provides a transparent framework for evaluating approximate unlearning. While currently optimized for 10-class image classification, future work will focus on scaling the system to handle the complexity of Large Language Models (LLMs) and expanding the attack simulations to include "white-box" scenarios where attackers have direct access to model weights.

Summary updated

BibTeX
@article{lee2026unlearning,
  author = {Jaeung Lee and Suhyeon Yu and Yurim Jang and Simon S. Woo and Jaemin Jo},
  title = {{Unlearning Comparator: A Visual Analytics System for Comparative Evaluation of Machine Unlearning Methods}},
  journal = {IEEE Trans. Vis. Comput. Graph.},
  year = {2026},
  doi = {10.1109/tvcg.2026.3658325},
}
Figure