Disentangling Forgetting from Utility: Lightweight LLM Unlearning via Orthogonal Null-Space Projection

Xiuyuan Wang, Weiming Liu, Zongxin Yang, Fan Wang, Liemiao Jin, Rubin Qin, Wei Chen, Bin Shi, Chaochao Chen, Xiaolin ZhengarXiv 2026 · arXiv

TL;DR

UUE (Untargeted Unlearning via Null-space-guided Editing) is a lightweight framework that excises specific knowledge from LLMs while strictly preserving general utility. By projecting parameter updates into the orthogonal null space of retained knowledge and using a novel "untargeted" objective, it effectively removes hazardous or private information without the computational cost or performance degradation typical of fine-tuning.

What Problem Does This Paper Address?

Large Language Models (LLMs) are notorious for memorizing their training data, which often includes sensitive, private, or hazardous information. "Unlearning"—the process of removing specific knowledge after training—is essential for safety and privacy.

However, existing methods face three major hurdles:

  1. Utility Degradation (CH-1): Knowledge in LLMs is "entangled." Deleting one fact often inadvertently breaks the model's general reasoning or linguistic abilities.
  2. Computational Cost (CH-2): Full-parameter fine-tuning is prohibitively expensive for modern scale models.
  3. The "Targeted" Trap (CH-3): Most methods force the model to map "forget" data to a specific refusal response (e.g., "I don't know"). This can cause distributional distortion and often only suppresses the knowledge at the surface level rather than removing it.

Methods

The authors propose UUE, which reformulates unlearning as a controllable model editing task using two key innovations:

  • Lightweight Unlearning Adapters: Instead of modifying the frozen backbone, UUE inserts pluggable adapters after the Feed-Forward Networks (FFNs) in middle transformer blocks. This allows for efficient deployment and easy "reversal" by simply removing the adapter.
  • Orthogonal Null-Space Projection: To protect general utility, UUE calculates the "near-null" subspace of the model's representations for a retain set (general knowledge). By constraining adapter updates to this subspace, the authors ensure that the edits have minimal to no impact on the model's performance on unrelated tasks.
  • Untargeted Editing Objective: Rather than pulling the model toward a single refusal target, UUE pushes "forget-set" representations toward their antipodal points (the opposite direction in embedding space). This moves the model into a broad "safe region" without prescribing exactly what it should say, avoiding the brittle nature of targeted refusal.
Illustration of targeted vs. untargeted unlearning and the UUE framework.

The framework also includes UUE-L, a parameter-efficient variant that uses iterative low-rank optimization (similar to LoRA) for even greater scalability.

Datasets

The authors evaluated UUE on two primary benchmarks:

  • TOFU (Task of Fictitious Unlearning): A dataset of 200 synthetic author biographies used to measure how well a model can forget specific entities while retaining general knowledge.
  • WMDP (Weapons of Mass Destruction Proxy): A benchmark designed to measure the removal of hazardous knowledge related to biosecurity, cybersecurity, and chemical weapons.
  • WikiText: Used as the "retain set" to represent general-domain knowledge that must be preserved during the unlearning process.

Key Insights and Ideas

  • The Power of the Null Space: The most striking insight is that we can mathematically "shield" general knowledge. By using Eigenvalue Decomposition (EVD) on the Gram matrix of retain-set representations, UUE identifies directions in the parameter space that the model doesn't use for general tasks, using those directions exclusively for unlearning.
  • Middle Layers are the "Sweet Spot": The authors found that editing middle layers (e.g., layers 10–15 in a 24-layer model) provides the best balance. Early layers propagate changes too widely, while late layers lack the semantic "leverage" to effectively excise deep-seated knowledge.
  • Closed-Form Efficiency: Because the UUE objective is formulated as a constrained quadratic problem, it yields a closed-form analytical solution. This means the final adapter update requires no iterative training, making it incredibly fast.

Key Results

  • Superior Forgetting: On the TOFU benchmark, UUE achieved significantly higher "Forget Quality" (FQ) scores than baselines like Gradient Ascent (GA) and DPO, while maintaining higher "Model Utility" (MU).
  • Hazardous Knowledge Removal: On WMDP, UUE reduced accuracy on hazardous questions to near-random levels (approx. 25%) while preserving substantially more general utility (MMLU scores) compared to aggressive baselines like RMU.
  • Speed: As shown in the execution time analysis, UUE and UUE-L are faster than almost all competing methods. The closed-form update accounts for only ~5.5% of the total runtime, with the bottleneck being the initial subspace construction.
Execution time comparison on TOFU forget01 with Phi-1.5B.

Limitations and Future Work

  • Retain Set Dependency: The effectiveness of the null-space projection depends on having a representative retain set. If the retain set is too small or narrow, the "shield" may be incomplete.
  • Response Quality: While UUE successfully prevents the model from reproducing forgotten facts, the resulting "untargeted" responses can sometimes be incoherent or incomplete rather than a polite refusal.
  • Future Directions: The authors suggest exploring how to combine UUE with better response-generation techniques to ensure that once a fact is forgotten, the model still produces high-quality (albeit safe) text.

Summary updated

Abstract

Large Language Models (LLMs) pose significant security risks by retaining private or hazardous information, creating an urgent need for unlearning techniques that excise specific knowledge without compromising general utility. However, prevailing fine-tuning-based unlearning methods struggle to achieve this balance, often suffering from computational inefficiency and unintended utility degradation due to the entangled nature of knowledge within model parameters. To disentangle forgetting from utility, we propose UUE, a theoretically grounded framework that reformulates unlearning as controllable model editing. First, UUE projects parameter updates into the orthogonal left null space of retained knowledge representations, strictly ensuring that the editing process causes minimal interference to the model's general capabilities. Second, distinct from targeted mapping, we introduce a convex untargeted objective that derives a closed-form analytical solution to effectively disrupt undesired knowledge by pushing forget-set embeddings toward their antipodal points. We implement UUE via lightweight, pluggable adapters (including a low-rank variant, UUE-L), enabling efficient deployment without modifying the frozen backbone. Extensive experiments on TOFU and WMDP benchmarks demonstrate that UUE and UUE-L achieve superior unlearning efficacy, significantly outperforming existing methods.
BibTeX
@misc{wang2026disentangling,
  author = {Xiuyuan Wang and Weiming Liu and Zongxin Yang and Fan Wang and Liemiao Jin and Rubin Qin and Wei Chen and Bin Shi and Chaochao Chen and Xiaolin Zheng},
  title = {{Disentangling Forgetting from Utility: Lightweight LLM Unlearning via Orthogonal Null-Space Projection}},
  year = {2026},
  eprint = {2611.17864},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17864},
}
Figure