OrtSAE: Orthogonal Sparse Autoencoders Uncover Atomic Features

Anton Korznikov, Andrey V. Galichin, Alexey Dontsov, Oleg Rogov, Elena Tutubalina, Ivan OseledetsSubmission to ICLR 2026 · Desk rejected

TL;DR

Current Sparse Autoencoders (SAEs) frequently merge distinct concepts into single latents (feature composition) or allow specific features to swallow general ones (feature absorption). OrtSAE (Orthogonal SAE) tackles this by adding a scalable, chunk-wise orthogonality penalty to the decoder features during training. Without altering the underlying architecture, OrtSAE reduces feature absorption by 65% and composition by 15% while discovering more unique, atomic features with negligible computational overhead.

What Problem Does This Paper Address?

In mechanistic interpretability, researchers use SAEs to disentangle the messy, polysemantic activations of Large Language Models (LLMs) into interpretable, monosemantic concepts. However, existing SAE objectives (optimizing reconstruction error against an $L_0$ or $L_1$ sparsity constraint) suffer from two major pathologies:

  1. Feature Absorption: A narrow, token-specific latent absorbs a broad feature to save activation budget. For example, a single "elephant" latent might absorb the "starts with E" concept, leaving the general feature blind to the word "elephant."
  2. Feature Composition: Independent atomic concepts that frequently co-occur (e.g., "red" and "square") collapse into a single composite representation ("red square").

Recent hierarchical fixes (like Matryoshka SAEs) add substantial training overhead and suffer from feature hedging (blurring correlated concepts at higher levels). OrtSAE addresses this core failure mode directly: if true generative features in superposition should be nearly orthogonal, penalizing high cosine similarity among dictionary latents should force them to stay separate and atomic.

Methods

OrtSAE introduces a global orthogonality regularizer to the standard SAE reconstruction and sparsity loss:

  • Chunk-Wise Orthogonality Penalty: Calculating all pairwise cosine similarities across a large dictionary of $m$ latents scales quadratically ($O(m^2)$), which is computationally infeasible when $m = 65{,}536$. OrtSAE solves this by partitioning the dictionary into $K$ chunks (e.g., 8 chunks of 8,192 latents). Within each chunk, it identifies the maximum pairwise cosine similarity for each latent, squares it to heavily penalize high overlaps, and averages across chunks. This preserves an efficient $O(m)$ linear scaling.
  • Intermittent Loss Updates: To further accelerate training, the orthogonality loss can be computed only once every 5 iterations (scaling the penalty weight $\gamma$ by $5\times$). This keeps training time within 4% of a standard BatchTopK SAE baseline.
  • Geometric Regularization: Rather than hard-constraining the basis, OrtSAE applies a soft penalty. This allows the model to retain genuinely non-orthogonal concepts (like circular representations for days of the week) if the reconstruction loss strongly demands it.

Datasets

  • Training Corpus: 500 million tokens sampled from OpenWebText with a 1,024-token context window.
  • LLM Activations: Extracted from Gemma-2-2B (layers 12 and 20) and Llama-3-8B (layer 20).
  • Downstream Benchmark: SAEBench, evaluated on Spurious Correlation Removal (SCR), Targeted Probe Perturbation (TPP), Sparse Probing, and RAVEL (Resolving Attribute–Value Entanglements in Language Models).
  • Synthetic Dataset: A synthetic setup ($n=100$, $m=3{,}200$) with ground-truth features sampled uniformly on a unit sphere and activated via heavy-tailed Bernoulli draws to verify atomic feature recovery.

Key Insights and Ideas

  • Global vs. Local Orthogonality Matters: Alternative methods like Matching Pursuit SAE (MP-SAE) enforce local orthogonality only among features co-activating on the same token. The authors demonstrate that this actually worsens feature composition (composition rate rises to 0.474), because the sequential residual pursuit incentivizes the model to learn composite "shortcuts." In contrast, OrtSAE’s global dictionary penalty naturally drives down both global and local interference (lower Babel coherence scores).
  • High Similarity is Mathematically Inherent to Merging: The authors derive that whenever a composite feature forms from two nearly orthogonal atomic features, its cosine similarity to each atomic feature jumps from roughly $\approx 0$ to $\approx 0.70$. Penalizing high cosine similarity targets the geometric signature of absorption and composition directly at its root.

Key Results

  • Higher Atomicity: At an $L_0$ sparsity of 70 on Gemma-2-2B, OrtSAE reduces feature absorption by 65% and feature composition by 15% compared to traditional SAEs.
  • Dictionary Uniqueness: OrtSAE discovers 9% unique features (features with cross-model cosine similarity $< 0.2$ relative to BatchTopK), compared to only 1.5% found by BatchTopK—a six-fold increase.
  • Downstream Tasks: On SAEBench, OrtSAE improves Spurious Correlation Removal (SCR) by +6%, as isolated atomic latents can be zero-ablated without collateral damage to unrelated concepts. Performance on Sparse Probing and RAVEL remains on par with top baselines.
  • Near-Zero Training Tax: OrtSAE matches standard BatchTopK reconstruction fidelity (explained variance within ~1%) while adding only a 4% to 11% runtime increase, far outperforming MP-SAE (which is 50–100$\times$ slower).

Limitations and Future Work

  • Non-Orthogonal Geometries: While rare, certain conceptual domains (such as circular representations of months or compass directions) are non-orthogonal by nature. Over-penalizing similarity could potentially distort these specific structures.
  • Gap Behind Hierarchical Models on Some Benchmarks: Although OrtSAE avoids the training cost and feature hedging of Matryoshka SAEs, Matryoshka still achieves slightly higher raw scores on certain multi-class ablation tasks (TPP).
  • Future Work: Applying OrtSAE's atomic primitives to downstream mechanistic interpretability workflows, particularly for uncovering cleaner, more human-auditable neural circuit graphs.

Summary updated

Abstract

Sparse autoencoders (SAEs) are a technique for sparse decomposition of neural network activations into human-interpretable features. However, current SAEs suffer from feature absorption, where specialized features capture instances of general features creating representation holes, and feature composition, where independent features merge into composite representations. In this work, we introduce Orthogonal SAE (OrtSAE), a novel approach aimed to mitigate these issues by enforcing orthogonality between the learned features. By implementing a new training procedure that penalizes high pairwise cosine similarity between SAE features, OrtSAE promotes the development of disentangled features while scaling linearly with the SAE size, avoiding significant computational overhead. We train OrtSAE across different models and layers and compare it with other methods. We find that OrtSAE discovers 9% more distinct features, reduces feature absorption (by 65%) and composition (by 15%), improves performance on spurious correlation removal (+6%), and achieves on-par performance for other downstream tasks compared to traditional SAEs.
BibTeX
@misc{korznikov2025ortsae,
  author = {Anton Korznikov and Andrey V. Galichin and Alexey Dontsov and Oleg Rogov and Elena Tutubalina and Ivan Oseledets},
  title = {{OrtSAE: Orthogonal Sparse Autoencoders Uncover Atomic Features}},
  year = {2025},
  eprint = {2509.22033},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.22033},
}
Figure