Orthogonal Gradient Projection for Machine Unlearning

57 concepts38 papers1 proposal

When a machine learning model must "forget" a specific subset of its training data—whether to comply with privacy regulations like the "right to be forgotten," remove copyrighted material, or excise harmful behaviors—the naive approach is to apply gradient ascent on the target data. However, this often triggers "over-unlearning," a phenomenon where the model's performance on the remaining data (the retain set) catastrophically degrades.

The core challenge of machine unlearning is an optimization conflict: how do you aggressively alter a model's behavior on one set of samples without destroying its generalized capabilities on everything else?

A rapidly growing body of research solves this problem through a geometric lens: Orthogonal Gradient Projection. By exploiting the high-dimensional geometry of neural networks, these methods isolate unlearning updates so they have zero first-order impact on the retained knowledge.

The Geometry of Non-Interference

To understand why orthogonal projection works, consider the parameter space of a neural network. When we want to maintain performance on a dataset, we want to avoid changing the parameters in any direction that increases the retention loss ($L_{\text{retain}}$). The direction of maximum performance degradation is defined by the retention gradient ($\nabla L_{\text{retain}}$).

If we update the model weights by a vector $\Delta w$ to force forgetting, the change in the retention loss can be approximated using a first-order Taylor expansion:

$$\Delta L_{\text{retain}} \approx \langle \nabla L_{\text{retain}}, \Delta w \rangle$$

If $\Delta w$ points in the opposite direction of $\nabla L_{\text{retain}}$, the loss increases, and the model suffers collateral damage. However, if we can constrain $\Delta w$ such that it is perfectly perpendicular (orthogonal) to $\nabla L_{\text{retain}}$, their dot product becomes zero:

$$\langle \nabla L_{\text{retain}}, \Delta w \rangle = 0$$

In this idealized geometric setup, updating the weights along $\Delta w$ modifies the model's response to the forgotten data while leaving the retention loss entirely invariant. As noted by Melamed et al. in Provable Unlearning with Gradient Ascent on Two-Layer ReLU Neural Networks, the high-dimensional geometry of neural networks naturally facilitates this; the gradients of individual data points are often naturally near-orthogonal, meaning targeted unlearning can be achieved with minimal intrinsic interference.

       ▲ Gradient for Retention (∇L_retain)
       │
       │          Projected Unlearning Update (Δw)
       │         ┌──────────────────────────────►
       │         │  (Angle is 90 degrees:
       │         │   Dot product = 0)
       │         │
───────┼─────────┴───────────────────────────────
       │        /
       │       /
       │      /
       │     ▼ Original Unlearning Gradient (g_forget)

Implementing the Projection

Researchers have developed several distinct strategies to calculate and enforce these orthogonal constraints, operating either directly on parameter gradients or on internal feature representations.

1. Gradient-Level Orthogonalization

The most direct application of this concept is to modify the unlearning gradient before it updates the weights. In Go Beyond Your Means: Unlearning with Per-Sample Gradient Orthogonalization, Shamsian et al. introduce OrthoGrad, which computes the unlearning gradient and projects it onto the orthogonal complement of the individual gradients of the retain samples. This ensures that the unlearning trajectory avoids any direction that would disturb the retained data.

When conflicts are detected dynamically, Wang et al. propose Gradient Rectified Unlearning (GRU) in GRU: Mitigating the Trade-off between Unlearning and Retention for LLMs. GRU measures the cosine similarity between unlearning updates and retention gradients; if they point in opposing directions, the unlearning gradient is rectified via orthogonal projection to eliminate destructive interference.

This framework even holds true in overparameterized models where standard loss gradients vanish because the model perfectly fits the data. In Machine Unlearning under Overparameterization, Block et al. demonstrate that restricting parameter updates to the null space of the retained dataset's gradients allows the model to shift its parameters to forget target data while remaining strictly on the manifold of optimal solutions for the remaining data.

2. Feature-Space and Null-Space Calibration

Rather than working purely with loss gradients, another faction of work targets the model's internal representations. The intuition here is to find the mathematical "null space" of the features we want to keep, ensuring that any weight modifications occur entirely within a subspace that the retained data does not utilize.

In Machine Unlearning via Null Space Calibration, Chen et al. confine weight updates to the null space of the remaining data's feature representations, preventing the "over-unlearning" effect entirely. Similarly, Rana et al. develop Projected-Gradient Unlearning (PGU) in Learn to Unlearn for Deep Neural Networks: Minimizing Unlearning Interference with Gradient Projection, projecting updates into a subspace orthogonal to the core features essential for the retained data.

This feature-space approach is powerful enough to operate even when the remaining data is completely unavailable due to strict privacy constraints. In Zero-Shot Machine Unlearning with Proxy Adversarial Data Generation, Xin Yu generates adversarial proxy data to map out the feature subspace occupied by the remaining data, allowing the algorithm to restrict updates to its orthogonal complement without ever seeing the original benign dataset.

3. Bidirectional Gradient Surgery

In complex generative models, unlearning is rarely a simple one-way constraint; instead, the forgetting objective and the utility preservation objective actively compete. In these scenarios, researchers use mutual projection techniques to harmonize the optimization process.

For text-to-image models, Ko et al. design a "Restricted Gradient" framework in Boosting Alignment for Post-Unlearning Text-to-Image Generative Models. When the objectives produce conflicting gradients, they apply gradient surgery to project each gradient onto the normal vector of the other. This mutual orthogonalization ensures that the model updates result in simultaneous, monotonic improvement across both tasks.

Conflicting Gradients:             Mutual Projection (Surgery):

        ▲ g_retain                         ▲ g_retain
        │                                  │
        │                                  │   g_forget_projected
        │                                  ├───► (Orthogonal to g_retain)
   ◄────┘                                  │
g_forget                                   │

Similarly, Huang et al. leverage meta-learning in Learning to Unlearn for Robust Machine Unlearning to project the forgetting gradient onto a subspace orthogonal to the retention gradient, preventing competing objectives from canceling each other out and stabilizing convergence.

The Limits of Orthogonality

While orthogonal projection elegantly bypasses the trade-off between forgetting and remembering, it is subject to a fundamental geometric constraint: subspace overlap.

The entire paradigm relies on the assumption that the features or gradients of the forget set and the retain set are sufficiently distinct. As Shamsian et al. explicitly warn in Go Beyond Your Means: Unlearning with Per-Sample Gradient Orthogonalization, if the gradient vector for unlearning a specific data point happens to be highly aligned with the gradients needed to maintain performance on the retain set, projecting it onto the orthogonal complement will shrink the vector toward zero.

In cases of severe entanglement—where the concept to be forgotten is deeply woven into the utility of the data to be kept—orthogonal projection can inadvertently eliminate the unlearning signal altogether. Resolving this entanglement remains a primary frontier for the field.

Key Foundational Work

Learning to Unlearn for Robust Machine Unlearning (2024) — contributed 2 concept elements to this direction.
Learn to Unlearn for Deep Neural Networks: Minimizing Unlearning Interference with Gradient Projection (2024) — contributed 2 concept elements to this direction.
Boosting Alignment for Post-Unlearning Text-to-Image Generative Models (2024) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_orthogonal_gradient_projection_conflicti_2bacf1a9,
  title = {orthogonal gradient projection (conflicting gradient projection)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/orthogonal-gradient-projection-conflicting-gradient-projection/rdr_2bacf1a944b0414e9d08bc8837000b98},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_orthogonal_gradient_projection_conflicti_2bacf1a9_v1,
  title = {orthogonal gradient projection (conflicting gradient projection)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/orthogonal-gradient-projection-conflicting-gradient-projection/rdr_2bacf1a944b0414e9d08bc8837000b98/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake