Spectral Unlearning: Transformer Structure-Preserving Updates for Language Models

Sung Il Choi, Junhao Cai, Dohun Kim, Changhee JooarXiv 2026 · arXiv

TL;DR

Most machine unlearning methods focus on what to optimize (the loss function) while ignoring how to update the model (the optimizer). This paper introduces Spectral Unlearning, a framework that replaces standard scalar-based updates with matrix-structured ones that respect the intrinsic geometry of Transformers. By aligning the update direction with the model's architecture, Spectral Unlearning removes targeted knowledge more effectively and efficiently than standard methods.

What Problem Does This Paper Address?

Large Language Models (LLMs) are notorious for memorizing sensitive information, from private user data to hazardous biological instructions. "Machine unlearning" aims to surgically remove this specific knowledge without forcing a full retraining of the model from scratch.

Current unlearning research focuses almost entirely on designing better "forget" and "retain" loss functions. However, when it comes to actually updating the model's weights, researchers typically fall back on standard optimizers like AdamW. These optimizers are "entry-wise," meaning they treat every single number in a weight matrix as an independent scalar.

The authors argue this is a mistake. Transformer weights are matrices that act on normalized hidden states. Treating them as a flat list of independent numbers ignores the correlations between parameters and the structural "geometry" of the network. This mismatch leads to inefficient updates that can either fail to forget the target data or accidentally destroy the model's general utility.

Methods

The authors propose Spectral Unlearning (SU). Instead of updating weights coordinate-by-coordinate, SU updates the entries of each Transformer weight matrix jointly.

  1. The Theoretical Foundation: The authors prove that for "pre-norm" Transformers (the standard architecture today), the natural way to measure the size of a weight update is the "RMS-to-RMS" norm. They show that the steepest descent direction under this norm is the spectral direction, which is derived from the Singular Value Decomposition (SVD) of the gradient.
  2. Practical Approximations: Computing a full SVD for every weight matrix at every step is computationally impossible for LLMs. To make this practical, the authors adapt three existing matrix-structured optimization techniques:
  3. Muon: Uses a Newton-Schulz iteration to approximate the spectral direction.
  4. KLShampoo: Maintains running estimates of gradient statistics to capture matrix structure.
  5. SOAP: Performs updates in the eigenbasis of the gradient covariance, which the authors find to be the most effective approximation.
Figure 1: The Llama-3.2-1B model forgets data from the TOFU benchmark through the negative preference optimization (NPO) unlearning method. Loss curves are plotted against (a) unlearning steps and (b) wall-clock time. Dashed curves use conventional entry-wise updates. Solid curves use our proposed spectral update (SU), realized through Muon, KLShampoo, and SOAP.

Datasets

The framework was evaluated using several prominent unlearning benchmarks:

  • TOFU (Task of Fictitious Unlearning): A dataset of fake author biographies used to measure how well a model can "forget" specific facts.
  • RWKU (Real-World Knowledge Unlearning): Focuses on removing knowledge about real-world famous people.
  • WMDP (Weaponized Model Detection and Prevention): A high-stakes benchmark for removing hazardous knowledge related to biosecurity and cybersecurity.

The authors tested these on a variety of models, including Llama-3.2 (1B and 3B), Llama-3 (8B), and Zephyr-7B.

Key Insights and Ideas

  • Geometry is a Design Choice: The paper highlights that the direction of an update is determined by the geometry we assume on the parameter space. By choosing a geometry that matches the Transformer's architecture, we get "cleaner" updates.
  • Capturing Correlations: Standard optimizers have near-zero "cosine similarity" with the true gradient covariance of the model. In contrast, matrix-structured optimizers like SOAP capture these correlations effectively.
Figure 2: Cosine similarity between the gradient covariance estimated by update rule and the true covariance during GD unlearning on Llama-3.2-1B. Similarity is measured on the weight matrices of (a) the attention output projection, (b) the MLP up-projection, and (c) the MLP down-projection.
  • Selectivity through Preconditioning: Spectral updates appear to be more "selective." They concentrate changes in the specific layers and blocks where the "forget" and "retain" signals disagree most strongly, leading to a better trade-off between removing data and keeping the model smart.

Key Results

  • Closing the Gap: On the TOFU benchmark, Spectral Unlearning (specifically using the SOAP variant) consistently outperformed baselines. In some cases, it made the unlearned model nearly indistinguishable from a model that had never seen the forget data at all.
  • Efficiency: As shown in Figure 1, spectral updates reach the target loss in significantly fewer steps and less total wall-clock time than AdamW, despite the extra math required per step.
  • Robustness: One of the most practical findings is that Spectral Unlearning is much less sensitive to the learning rate. While AdamW often "collapses" (the model becomes useless) if the learning rate is slightly too high, spectral methods degrade gracefully.
  • Resistance to Relearning: Models unlearned with spectral updates were harder to "attack." When researchers tried to force the model to relearn the forgotten data, the spectral-unlearned models resisted the recovery of that knowledge better than those updated with AdamW.

Limitations and Future Work

  • Computational Overhead: While faster in terms of total time to reach a goal, spectral updates do use more memory and more compute per individual step due to preconditioner maintenance.
  • Scale: The study focused on models up to 8B parameters. Whether these advantages hold at "frontier" scales (e.g., 70B or 400B parameters) remains an open question.
  • General Optimization vs. Unlearning: The authors note that spectral updates also improve standard training. Future work is needed to disentangle how much of the benefit is specific to the "unlearning" task versus just being a generally superior way to train Transformers.

Summary updated

Abstract

Machine unlearning in large language models (LLMs) aims to remove specific learned knowledge, capabilities, or behaviors while preserving model utility. Existing methods have mostly focused on the design of unlearning objectives, while largely overlooking the intrinsic matrix structure of transformer weights. In particular, standard optimizers operate on independent scalar coordinates, thereby ignoring this matrix structure. We propose spectral unlearning, an update framework that combines standard unlearning objectives with matrix-structured optimization. We show that the spectral update corresponds to the steepest-descent direction for transformers composed of linear layers with normalized inputs and outputs. Our approach is applied on top of existing unlearning methods and evaluated across complementary axes of memorization, privacy, and utility, summarized via their harmonic mean. Spectral unlearning consistently outperforms a broad range of baselines and closes most of the gap to a retain-trained reference. Code is available at https://github.com/unlearn-ml/spectral-unlearning/.
BibTeX
@misc{choi2026spectral,
  author = {Sung Il Choi and Junhao Cai and Dohun Kim and Changhee Joo},
  title = {{Spectral Unlearning: Transformer Structure-Preserving Updates for Language Models}},
  year = {2026},
  eprint = {2611.11902},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.11902},
}
Figure