Machine learning models are often "sticky"—once they see a piece of data, its influence is baked into millions of parameters. When a user exercises their "right to be forgotten," or when a developer discovers a malicious backdoor, the traditional solution is to retrain the entire model from scratch. This is the "gold standard" for privacy, but it is computationally ruinous for modern architectures.
Recent research has converged on a more efficient alternative: Approximate Machine Unlearning. Instead of starting over, these methods perform "surgery" on a pre-trained model to excise specific information while preserving general utility. This is achieved by manipulating the model’s internal geometry, its structural sparsity, and its gradient trajectories.
Geometric Repositioning: Pushing Data Away
One of the most intuitive ways to unlearn is to treat the model's latent space as a physical map. In a well-trained classifier, data points from the same class cluster together around a "centroid." To make a model forget a specific sample, we can physically move its representation.
In DUCK: Distance-based Unlearning via Centroid Kinematics, Jacopo Bonato and colleagues demonstrate that pushing a target sample’s embedding toward the nearest incorrect class centroid effectively erases its influence. This "kinematic" approach is significantly faster than retraining because it focuses on local geometric relationships rather than global weight optimization.
Similarly, Zhehao Huang et al. propose in Unified Gradient-Based Machine Unlearning with Remain Geometry Enhancement that unlearning can be viewed as optimization on a manifold defined by the "remain" data. By using a "fast-slow" update strategy, they account for the second-order geometry of this manifold, allowing generative models to forget specific classes in a fraction of the usual time.
Structural Surgery: Pruning and Sensitivity
If geometry is about where data lives in the latent space, structural methods are about which "wires" in the neural network carry that information.
Jinghan Jia and Jiancheng Liu argue in Model Sparsity Can Simplify Machine Unlearning that model sparsity is a secret weapon for unlearning. By pruning a model before unlearning, researchers can isolate the influence of specific data points. Sparsity simplifies the loss landscape, making it easier for approximate algorithms to reach the same state that a fully retrained model would occupy.
Taking this "surgery" metaphor further, Jing Wu and Mehrtash Harandi introduced Scissorhands. This framework identifies the specific network connections most sensitive to the data being forgotten—a "targeted lobotomy"—and then repairs the remaining structure to ensure the model doesn't lose its general intelligence.
Gradient Safeguards: Pivoting and Projection
The central challenge of unlearning is the "trilemma" between privacy (forgetting the target), utility (keeping the rest), and efficiency. To navigate this, researchers use gradient manipulation to ensure that the "forgetting" process doesn't accidentally damage the "remembering" process.
- Gradient Pivoting: Youngsik Hwang and Dong-Young Lim developed CUP, which treats unlearning as a multi-objective optimization problem. By "pivoting" the gradient, they allow practitioners to tune an "unlearning intensity" parameter, finding the perfect balance between privacy and performance.
- Orthogonal Projection: In "zero-shot" scenarios where the original training data is unavailable, Huiqiang Chen and colleagues use ZS-PAG to generate adversarial proxy data. They then use orthogonal gradient projection to ensure that the updates intended to erase the target data do not interfere with the subspaces containing the model's general knowledge.
The Frontier: From Vision to Language
While these geometric and structural mechanisms have proven highly effective in computer vision—as seen in Attribute Unlearning by Song Guo and CoUn by Yasser H. Khalil—a recurring theme across the field is the "modality gap."
Most current mechanisms rely on the continuous, Euclidean nature of image embeddings. It remains an open question whether these same principles of centroid kinematics or manifold optimization will translate to the discrete tokens and temporal dependencies of Large Language Models or the non-Euclidean structures of Graph Neural Networks. As unlearning moves from simple classification to complex reasoning, the "surgery" will likely need to become even more precise.