SVD as a Fast Interpretability Method for Transformers

Min Xue, Artur AndrzejakICML 2026 · OpenReview

Introduction: The Bottlenecks of Mechanistic Interpretability

Mechanistic interpretability is the quest to reverse-engineer neural networks and understand the "why" behind their outputs. Currently, the dominant approach relies on training auxiliary proxy models, such as Sparse Autoencoders (SAEs) or Cross-Layer Transcoders, to decompose complex internal activations into simpler, interpretable features.

However, these post-hoc approaches introduce two major bottlenecks:

  1. Computational Expense: Training an SAE can be nearly as resource-intensive as training the original model itself.
  2. Approximation Bias: Because SAEs are proxies attempting to mimic the original model, they inherently introduce approximation errors. Researchers risk interpreting the quirks of the SAE rather than the true logic of the underlying model.

To address this, researchers have introduced Native Network Anatomy (NaNA), a training-free interpretability framework that directly exploits the Singular Value Decomposition (SVD) of weight matrices in Transformer Multi-Layer Perceptron (MLP) sublayers. By operating natively on model parameters, NaNA improves scalability while preserving perfect fidelity to the original weights.

The NaNA Framework: Cracking the Code with Linear Algebra

Instead of training a new model to interpret an existing one, NaNA leverages SVD to uncover interpretable structures already living inside the model's weights. The framework introduces two core concepts:

Detector-Effector Units (DEUs)

By decomposing an MLP weight matrix into orthogonal, rank-1 subspaces, the researchers discovered that the projection matrices naturally form pairs of vectors. Within each unit:

  • The Detector vector functions as a sensor, identifying specific patterns or concepts in the input.
  • The Effector vector modulates the output, encoding specific semantic meaning into the model's forward pass.

Subspace Contribution Analysis (SCA)

To understand which DEUs actually matter for a given prediction, the authors developed SCA. This diagnostic method quantifies the direct causal contribution of individual native subspaces. It filters for "causal bridges"—units that are both highly activated by the current input and strongly aligned with the target output token. If a unit is loud but does not point toward the final answer, SCA correctly identifies it as noise.

Experimental Setup

The researchers evaluated NaNA across the GPT-2 family of models (Small, Medium, Large, and XL). To rigorously test the method, they utilized three distinct evaluation sets:

  • Anchor Cases ($D_{achr}$): 20 curated examples covering syntax, factual retrieval, and reasoning.
  • Generalization Dataset ($D_{gene}$): 100 diverse prefixes to test how well the identified pathways hold up across different contexts.
  • Controlled Variation Sets: Datasets designed to isolate specific semantics (e.g., varying the subject of a sentence while keeping a target positive adjective like "happy" constant).

Key Findings and Results

1. Orders-of-Magnitude Efficiency Gains

Unlike SAEs, which become exponentially harder to manage as models grow, NaNA’s computational cost scales linearly. For a model like GPT-2 XL, extracting functional pathways with NaNA takes approximately 8.5 minutes on a single GPU. In contrast, achieving similar results with SAEs or Transcoders can take over 24 hours.

2. High Fidelity and Native Sparsity

While interpretability often assumes models must be forced to be sparse (as in SAEs), NaNA demonstrates that the most important functional pathways are naturally sparse when viewed through the lens of SVD. Reconstructing the model's MLP layers using only the top 20 subspaces identified by NaNA successfully recovered 97% to 99% of the model's top-1 prediction accuracy.

3. Causal Proof via Ablation and Amplification

The researchers proved the functional necessity of these units through causal interventions. "Ablating" (turning off) the top-ranked subspaces caused a total semantic collapse of the model's prediction. Conversely, "amplifying" a specific unit—such as one corresponding to positive qualities—could actively steer the model to predict words like "happy" or "sunny," even when they were not the original top choice.

Limitations and Future Directions

While NaNA presents a highly efficient alternative to learned proxy approaches, the study notes several areas for future exploration:

  • Architectural Scope: The current experiments focus on the GPT-2 family. Validating this approach on modern architectures, such as Llama-3 or Mixture-of-Experts (MoE) models, remains an open question.
  • MLP-Only Focus: Currently, NaNA analyzes only MLP layers. While MLPs store the bulk of a model's "knowledge," Attention heads act as the crucial routers of information. Extending SVD-based analysis to Attention mechanisms is a vital next step.
  • Labeling Bias: The method relies on an external LLM (like GPT-4o-mini) to automatically label the semantic meaning of a subspace. This introduces a potential risk of the interpreting model projecting its own biases onto the results.

Ultimately, SVD-based analyses like NaNA offer a highly scalable, faithful, and training-free path forward for mechanistic interpretability, proving that sometimes the best tools for understanding complex neural networks are classic linear algebra techniques.

Summary updated

Abstract

Mechanistic interpretability of Transformer models commonly relies on training auxiliary proxy models, such as Sparse Autoencoders or Cross-Layer Transcoders. While effective, these post-hoc approaches introduce approximation bias and incur substantial computational overhead. We propose an alternative, training-free interpretability framework that directly exploits the Singular Value Decomposition (SVD) of weight matrices in Transformer MLP sublayers. By operating natively on model parameters, our method improves scalability while preserving fidelity to the original weights. We show that the projection matrices of MLP sublayers admit a natural decomposition into orthogonal, interpretable rank-1 subspaces, which we term **Detector-Effector Units** (DEUs). Within each unit, a singular vector functions as a detector of input patterns and modulates a coupled effector vector that encodes output semantics. Building on this structure, we introduce **Subspace Contribution Analysis** (SCA), a diagnostic method that quantifies the direct causal contribution of individual native subspaces to model predictions. Experiments across the GPT-2 family demonstrate that our framework, **Native Network Anatomy** (NaNA), identifies dominant functional pathways with orders-of-magnitude efficiency gains over training-based interpretability baselines, while maintaining weight fidelity. Our results suggest that SVD-based analyses provide a scalable and faithful alternative to learned proxy approaches for mechanistic interpretability.
BibTeX
@inproceedings{xue2026svd,
  author = {Min Xue and Artur Andrzejak},
  title = {{SVD as a Fast Interpretability Method for Transformers}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=7tt8TwMjdJ},
}
Figure