When training a neural network, optimization is straightforward: we feed the model data and update its weights to minimize a single loss function. But what happens when we want the model to forget something it has already learned?
Whether removing copyrighted text, erasing private user data, or scrubbing harmful concepts from a generative model, the task of machine unlearning introduces a fundamental paradox. We are no longer optimizing for a single objective. Instead, we are forcing the model to balance two competing goals: maximizing error on the "forget set" while maintaining high accuracy on the "retain set."
Naively, one might combine these objectives into a single loss function using a weighted sum. However, this frequently results in a phenomenon colloquially known as "lobotomizing" the model—erasing the target data but severely damaging the model's general intelligence and utility.
Recent insights across the field reveal that this degradation is not an unavoidable tax of unlearning. Rather, it is a geometric failure mode known as gradient conflict.
The Geometry of the Tug-of-War
To understand why unlearning fails, we must look at the geometry of the weight updates. Let $g_f$ be the gradient vector calculated to induce forgetting, and let $g_r$ be the gradient vector calculated to maintain performance on retained data.
When these two vectors point in roughly the same direction, optimization is harmonious. But during unlearning, these vectors frequently exhibit an obtuse angle relative to one another ($g_f \cdot g_r < 0$). They directly oppose each other.
g_f (Forget Gradient)
^
\
\
+------> g_r (Retain Gradient)
/
/
v
Conflicting Update Direction
As Zibin Pan et al. demonstrate in Multi-Objective Large Language Model Unlearning, traditional weighted-sum approaches fail because they attempt to force a single linear combination of these opposing forces. This causes the model to oscillate wildly or take destructive update steps that collapse the parameter substructures holding general knowledge.
This gradient entanglement is particularly severe for complex or highly integrated samples. When a model fights itself during optimization, the unlearning signal typically overwhelms the retention signal, leading to catastrophic utility loss.
Paradigm 1: Geometric Projection and Subspace Orthogonalization
If the core issue is that the forgetting gradient interferes with the retention gradient, the most direct solution is to geometrically isolate the unlearning signal. If we can project $g_f$ into a subspace where it is completely orthogonal to $g_r$, the weight updates intended for forgetting will cause zero collateral damage to the retained knowledge.
g_f (Forget Gradient)
^
| .
| . Projected Update
| .
+-------> Orthogonal Subspace (Safe for Retain Set)
This geometric intuition forms the basis of several recent frameworks:
- Gradient Rectified Unlearning (GRU): Yue Wang et al. introduce this framework in GRU: Mitigating the Trade-off between Unlearning and Retention for LLMs. GRU explicitly identifies the directions where forgetting updates directly oppose general knowledge retention and projects the updates away from those conflicting zones.
- Per-Sample Orthogonalization (OrthoGrad): In scenarios where only a tiny fraction of the original training data is available to guide retention (the low-data regime), Aviv Shamsian et al. propose OrthoGrad in Go Beyond Your Means: Unlearning with Per-Sample Gradient Orthogonalization. OrthoGrad computes individual gradients for each available retain sample and projects the unlearning gradient into a subspace strictly orthogonal to them, ensuring precise data removal without utility degradation.
- Restricted Gradients in Generative Models: This geometric conflict also plagues text-to-image models. When trying to erase copyrighted styles or nudity, the model often loses its ability to follow unrelated prompts. To solve this, Myeongseob Ko et al. implement a "Restricted Gradient" framework in Boosting Alignment for Post-Unlearning Text-to-Image Generative Models, isolating the concept-erasing gradients from the text-image alignment pathways.
Paradigm 2: Multi-Objective and Bi-Level Optimization
Rather than modifying gradients post-hoc via projection, another school of thought reformulates the unlearning objective entirely, treating it as a specialized multi-objective or hierarchical optimization problem.
The authors of Learning to Unlearn while Retaining: Combating Gradient Conflicts in Machine Unlearning introduce the LUR framework, which focuses on implicitly aligning the forgetting and retaining gradients during the optimization process itself, preventing the emergence of opposing directional forces.
When a strict hierarchy is required—such as ensuring that privacy-violating data is absolutely erased while maximizing remaining utility—bi-level optimization becomes a powerful tool. Sadia Asif et al. propose this in OFMU: OPTIMIZATION-DRIVEN FRAMEWORK FOR MACHINE UNLEARNING. OFMU operates via a nested loop structure:
$$\min_{\theta} \mathcal{L}_{\text{retain}}(\theta) \quad \text{subject to} \quad \theta \in \arg\min_{\theta'} \mathcal{L}_{\text{forget}}(\theta')$$
By prioritizing forgetting in an inner loop and restoring model utility in an outer loop—while applying an explicit penalty to decorrelate conflicting gradients—OFMU prevents the optimization process from stalling or taking destructive paths.
This multitask perspective is even viable when the original training data is completely missing. In Efficient Source-free Unlearning via Energy-Guided Data Synthesis and Discrimination-Aware Multitask Optimization, Xiuyuan Wang et al. show that synthesizing high-quality proxy data allows researchers to set up a multitask optimization space that successfully navigates and resolves the contradictory gradient paths of source-free unlearning.
Paradigm 3: Decoupling Optimizer History and States
A subtle but severe driver of gradient conflict occurs within the optimizer itself. Modern stateful optimizers like Adam do not just apply the current gradient; they accumulate historical running averages of gradients and their squares (momentum and variance).
When a model transitions from standard training to an unlearning regime, these accumulated optimizer states become heavily corrupted. The historical states tuned for retention clash violently with the sudden, sharp gradients required for forgetting.
To break this internal "tug-of-war," Xuyang Zhong et al. introduce DualOptim+ in DualOptim+: Bridging Shared and Decoupled Optimizer States for Better Machine Unlearning in Large Language Models. Instead of forcing a single optimizer state to track conflicting goals, DualOptim+ maintains a shared "base state" for core knowledge and decoupled "delta states" for task-specific residuals, adaptively balancing the historical updates.
+-----------------------+
| Shared Base State |
| (General Knowledge) |
+-----------+-----------+
|
+---------------+---------------+
| |
v v
+-----------------------+ +-----------------------+
| Decoupled Delta State | | Decoupled Delta State |
| (Forgetting Residual) | | (Retention Residual) |
+-----------------------+ +-----------------------+
Similarly, this decoupling can be applied to the model's internal representations. Han Yan et al. develop PRISM in Dual-Space Smoothness for Robust and Balanced LLM Unlearning, which enforces smoothness constraints simultaneously across both the internal representation space and the parameter space. This dual-space approach widens the margin between harmful and safe content, effectively decoupling the underlying gradient vectors.
Architecture-Specific Challenges
The manifestation of gradient conflict depends heavily on the model architecture and how data is presented to it.
Diffusion Models
In text-to-image diffusion models, the unlearning process often causes the model to oscillate between erasing a target concept and maintaining overall image generation quality. To stabilize this path, Jing Wu et al. introduce EraseDiff across their work (Erasing Undesirable Influence in Diffusion Models and EraseDiff: Erasing Data Influence in Diffusion Models), framing unlearning as a tightly constrained optimization problem that prevents these destructive oscillations.
Large Language Models and Mixed Prompts
In LLMs, gradient conflict can be exacerbated by prompt structure. Wonje Jeung et al. study this in SEPS: A Separability Measure for Robust Unlearning in LLMs. They discover that when a single input sequence contains interleaved "forget" and "retain" queries, preference-based unlearning suffers severe optimization failures. The model is forced to simultaneously maximize and minimize token likelihoods within the exact same context window, creating an acute, localized gradient conflict.
Quantized Neural Networks
For models deployed on edge devices, unlearning must occur within highly constrained parameter spaces. In Forget by Uncertainty: Orthogonal Entropy Unlearning for Quantized Neural Networks, Tian Zhang et al. address the discrete weight spaces of Quantized Neural Networks (QNNs). Because standard continuous gradient alignment is impossible in QNNs, they introduce Orthogonal Entropy Unlearning to resolve directional conflicts within discrete boundaries, preventing the network from "misremembering" or corrupting adjacent quantized states.
The Role of the Loss Landscape
Finally, the geometry of the loss landscape itself dictates how severe these conflicts are. Standard gradient descent often lands models in sharp minima, where even a minor weight update causes massive changes in output behavior.
Haoran Tang et al. demonstrate in Sharpness-Aware Machine Unlearning that leveraging Sharpness-Aware Minimization (SAM) fundamentally alters this dynamic. By steering the model toward flatter minima for the retain set while intentionally overfitting and scrubbing the forget set via a "Sharp MinMax" strategy, they mitigate the destructive impact of gradient interference.
Conclusion
Machine unlearning is fundamentally a geometric problem. The historical tendency of unlearning algorithms to degrade model performance is not an intrinsic limitation of neural networks, but a consequence of gradient conflict. By treating unlearning as a multi-objective optimization problem—whether through orthogonal projections, bi-level optimization loops, or decoupled optimizer states—modern frameworks allow models to cleanly excise targeted data while keeping their broader capabilities entirely intact.
Key Foundational Work
Efficient Source-free Unlearning via Energy-Guided Data Synthesis and Discrimination-Aware Multitask Optimization (2025) — contributed 5 concept elements to this direction.
Learning to Unlearn while Retaining: Combating Gradient Conflicts in Machine Unlearning (2025) — contributed 4 concept elements to this direction.
EraseDiff: Erasing Data Influence in Diffusion Models (2024) — contributed 1 concept elements to this direction.