Lock on Target! Precision Unlearning via Directional Control

Yuntao Wen, Ruixiang Feng, Feng Guo, Yifan Wang, Ran Le, Yang Song, Shen Gao, Shuo ShangEMNLP 2025

TL;DR

This paper introduces DOGE, a novel method for "unlearning" specific knowledge from large language models (LLMs) with unprecedented precision. DOGE tackles the critical challenges of collateral forgetting (erasing unwanted data also erases related good data) and generality forgetting (unlearning degrades overall model performance) by identifying and leveraging a specific "unlearning direction" within the model's internal representations.

What Problem Does This Paper Address?

Large Language Models (LLMs) are incredibly powerful but can inadvertently learn harmful, sensitive, or outdated information from their vast training data. Removing this unwanted knowledge without costly retraining is crucial for model safety, privacy, and regulatory compliance (e.g., the "right to be forgotten"). However, existing unlearning methods struggle with two major issues:

  1. Collateral Forgetting: When trying to erase specific facts (e.g., a person's private address), the model might also forget related, desirable information (e.g., that person's published works).
  2. Generality Forgetting: Aggressive unlearning techniques can degrade the LLM's overall capabilities, making it perform worse on tasks it was never intended to forget.

These problems stem from the "imprecision" of current methods, which often make coarse-grained updates to the model's parameters, making it hard to pinpoint and modify only the specific knowledge to be removed.

Methods

DOGE (Direction Guided Unlearning) proposes a three-step approach to achieve precise unlearning:

  1. Unlearning Direction Extraction: This is the core innovation. DOGE analyzes the internal representations (activations) of the LLM for both "forget" samples (data to be removed) and "retain" samples (data to be kept). By comparing these representations, it identifies a specific "unlearning direction" in the model's parameter space that uniquely corresponds to the knowledge to be forgotten. This direction is found by looking at the differences in residual stream activations across layers.
  2. Orthogonal Intervention via Unlearning Direction: Once the unlearning direction is identified, DOGE uses it to precisely manipulate the model's internal representations. For forget samples, it amplifies components along this direction to reinforce the forgetting process. For retain samples, it suppresses components along this direction (by projecting onto its orthogonal complement) to preserve existing knowledge and prevent collateral damage.
  3. Direction Controlled Unlearning: The identified unlearning direction is then used to guide the actual parameter updates during the unlearning process. This ensures that updates are focused on the specific subspace associated with the unwanted knowledge, minimizing interference with retained information and general model capabilities.

Datasets

The paper evaluates DOGE on two widely used datasets for unlearning research:

  • TOFU (Task of Fictitious Unlearning): This dataset contains synthetic author profiles and question-answer pairs, with different subsets for forgetting, retaining, real authors, and world facts. It also includes various forgetting settings (1%, 5%, 10% of data to forget).
  • WMDP-cyber: This dataset consists of multiple-choice questions related to hazardous knowledge in biosecurity, cybersecurity, and chemical security.

Key Insights and Ideas

  • Directional Unlearning is Key: The central idea is that knowledge is encoded in specific directions within the LLM's high-dimensional parameter space. By identifying and targeting this "unlearning direction," we can achieve precise erasure.
  • Residual Stream Activations are Informative: The paper leverages residual stream activations, which have shown promise in distinguishing different model behaviors, to capture the subtle differences between forget and retain knowledge.
  • Balancing Act: DOGE effectively balances the trade-off between forgetting unwanted information and preserving useful knowledge and general capabilities, a common challenge in unlearning.
  • Interpretable Intervention: The orthogonal intervention mechanism provides a more interpretable way to control how unlearning affects model representations.

Key Results

  • Superior Forgetting Precision: DOGE achieves state-of-the-art performance in forgetting unwanted knowledge, demonstrated by the lowest ROUGE and Probability scores on the TOFU dataset's forget task.
  • Preservation of Retained Knowledge: DOGE excels at preserving retained knowledge, achieving the best Probability and Truth Ratio scores on the TOFU retain task, indicating minimal loss relative to the base model.
  • Maintained General Capabilities: The method shows competitive performance on general ability benchmarks (Real Author and Word Fact sets), suggesting it effectively mitigates generality forgetting.
  • Effectiveness on Sensitive Knowledge: On the WMDP-cyber dataset, DOGE successfully unlearned sensitive knowledge while maintaining strong general reasoning abilities, outperforming other methods in forgetting effectiveness.
  • Robustness: DOGE demonstrates robustness against jailbreaking attacks, as indicated by its PrivLeak score.

Limitations and Future Work

  • Data Dependency: The method relies on a clear distinction between forget and retain samples, which might not always be readily available in real-world scenarios.
  • Computational Overhead: Extracting the unlearning direction introduces some computational overhead, which could be a bottleneck for smaller research teams, although it's significantly less than full retraining.
  • Future Work: The authors suggest exploring broader implications of automated unlearning in sensitive or adversarial contexts and addressing the limitations of data availability.

Summary updated

BibTeX
@inproceedings{wen2025lock,
  author = {Yuntao Wen and Ruixiang Feng and Feng Guo and Yifan Wang and Ran Le and Yang Song and Shen Gao and Shuo Shang},
  title = {{Lock on Target! Precision Unlearning via Directional Control}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure