MCU: Improving Machine Unlearning through Mode Connectivity

Yingdan Shi, Ren Wang

Abstract

Machine Unlearning (MU) is essential for complying with data privacy regulations like GDPR, yet traditional methods often struggle with the trade-off between forgetting target data and maintaining model utility. While retraining from scratch is the gold standard, it is computationally prohibitive. Existing approximate methods typically rely on linear weight updates, which often fail due to the highly nonlinear nature of neural network loss landscapes. We propose Mode Connectivity Unlearning (MCU), a framework that identifies nonlinear "unlearning pathways" using quadratic Bézier curves. By integrating a parameter masking strategy and an adaptive penalty mechanism, MCU efficiently discovers a spectrum of models that achieve performance nearly identical to retraining. Our evaluations across diverse architectures and benchmarks demonstrate that MCU significantly improves the stability and effectiveness of machine unlearning.

Introduction

The goal of Machine Unlearning is to remove the influence of specific training data from a model without requiring a full retrain. A common approach involves "linear task arithmetic," where a forgetting vector is subtracted from the model's weights. However, this often leads to weight entanglement, where updates intended to forget specific data inadvertently degrade the model's performance on the remaining data.

MCU addresses this by moving beyond point-to-point linear updates. Instead, it searches for a curved path in the parameter space where high-quality unlearning models reside, allowing for a more nuanced balance between privacy and utility.

Methodology

Overview of the MCU framework, illustrating parameter masking, nonlinear pathway searching, and model region identification.

The MCU framework consists of three primary innovations:

  1. Nonlinear Pathway Searching: MCU utilizes a quadratic Bézier curve to connect the original model ($\theta_o$) and a "pre-unlearning" model ($\theta_p$) generated by an existing MU method. By optimizing a "control model" ($\theta_c$), the framework identifies a curved trajectory in the parameter space that contains a spectrum of effective unlearning solutions.
  2. Parameter Mask Strategy: To focus the unlearning process and reduce computational costs, MCU identifies parameters critical for retention versus those critical for forgetting. It employs a "filter and reserve" approach to create a binary mask, freezing retention-critical weights. This strategy results in a significant 75.23% speedup in backward propagation.
  3. Adaptive Unlearning Penalty ($\beta$): Rather than relying on static hyperparameters, MCU introduces an adaptive strategy that automatically adjusts the penalty coefficient ($\beta$) during training. This ensures the model maintains an optimal balance between forgetting the target data and preserving accuracy on the retained set.

Experimental Evaluation

We evaluated MCU on three standard image classification benchmarks using diverse architectures:

  • CIFAR-10 with PreResNet-110.
  • ImageNet-100 with Vision Transformer (ViT).
  • Tiny-ImageNet with VGG-16-BN.

The framework was tested under two primary scenarios: random data forgetting (removing a subset of samples) and class-wise forgetting (removing an entire category).

Key Results and Insights

Accuracy spectrum along the discovered pathway for random and class-wise forgetting.
  • Discovery of Model Spectrums: Unlike traditional methods that produce a single model, MCU uncovers a continuous spectrum of models along the path $t \in [0, 1]$. This allows practitioners to select a model that best fits their specific privacy and performance requirements.
  • Superior Accuracy Alignment: Across all datasets, the adaptive version of MCU ($MCU_\beta$) achieved the lowest average gap compared to the retraining (RT) baseline. In many instances, the performance was virtually indistinguishable from a model retrained from scratch.
  • Efficiency and Robustness: Despite the added complexity of searching a nonlinear path, the parameter masking strategy keeps runtime competitive. Furthermore, MCU proved robust even when only a small fraction (e.g., 30%) of the original retaining data was available for the unlearning process.
  • Privacy Protection: Membership Inference Attack (MIA) metrics confirmed that MCU effectively erases the influence of forgotten data, making it nearly impossible for attackers to identify if a specific sample was part of the original training set.

Conclusion

MCU represents a significant step forward in Machine Unlearning by leveraging the concept of mode connectivity. By moving from linear updates to nonlinear pathways, the framework provides a more stable and effective means of data removal. Future work may explore higher-order Bézier curves or surfaces to connect multiple models, potentially revealing even more optimal regions for unlearning in complex loss landscapes.

Summary updated

BibTeX
@inproceedings{shi2026mcu,
  author = {Yingdan Shi and Ren Wang},
  title = {{MCU: Improving Machine Unlearning through Mode Connectivity}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=SaUgHTz5Yo},
}
Figure