OrtSAE: Orthogonal Sparse Autoencoders Uncover Atomic Features

Anton Korznikov, Andrey V. Galichin, Alexey Dontsov, Oleg Rogov, Elena Tutubalina, Ivan V. Oseledets

Abstract

Sparse autoencoders (SAEs) are a critical technique for decomposing neural network activations into human-interpretable features. However, current SAEs often suffer from "identity crises": feature absorption, where general features are swallowed by specific ones, and feature composition, where independent concepts merge into a single representation. We introduce OrtSAE (Orthogonal SAE), a novel approach that mitigates these issues by enforcing geometric orthogonality between learned features. By implementing a computationally efficient training penalty, OrtSAE reduces feature absorption by 65% and composition by 15%, discovering 9% more unique, "atomic" features while maintaining high reconstruction fidelity.

The Challenge of Feature Identity

Mechanistic interpretability aims to identify the fundamental "atoms" of thought within Large Language Models (LLMs). While SAEs are the current standard for this task, they frequently fail in two ways:

  1. Feature Absorption: A broad concept (e.g., "words starting with E") is captured by multiple specific features (e.g., "elephant", "eagle"), leaving a "hole" in the representation of the general concept.
  2. Feature Composition: Distinct concepts (e.g., "red" and "square") are merged into a single composite feature ("red square"), hindering fine-grained analysis.

These failures occur because standard SAE objectives prioritize reconstruction and sparsity without explicitly discouraging feature overlap in the vector space.

Methodology: Enforcing Geometric Atomicity

OrtSAE addresses these issues by adding an orthogonality penalty to the standard SAE loss function. This penalty specifically targets and punishes high pairwise cosine similarity between the decoder feature vectors, forcing the model to find non-overlapping directions in the activation space.

Distribution of pairwise cosine similarity between SAE features. OrtSAE (red) shows significantly lower similarity compared to BatchTopK (green) and ReLU (blue) baselines.

To ensure this approach scales to massive models, the authors implemented two key optimizations:

  • Chunk-wise Strategy: The latent space is randomly partitioned into smaller chunks, and the penalty is only computed within these chunks. This reduces computational complexity from quadratic to linear ($O(m)$).
  • Sparse Computation: The orthogonality loss is calculated only every fifth training iteration, resulting in a negligible training overhead of approximately 4% compared to standard baselines.

Key Results and Insights

We evaluated OrtSAE on Gemma-2-2B and Llama-3-8B using the SAEBench suite. The results demonstrate that enforcing geometric orthogonality directly translates to conceptual "atomicity."

Summary of OrtSAE performance across key metrics: Explained Variance, Absorption rate, Composition rate, and Cosine Similarity distribution.

Reducing Failure Modes

As shown in the summary metrics, OrtSAE significantly outperforms traditional SAEs in reducing both absorption (-65%) and composition (-15%). By preventing features from "hiding" inside one another, OrtSAE discovered 9% more unique features compared to the BatchTopK baseline. Qualitative analysis supports this; for instance, OrtSAE successfully decomposed the token "Queen" into a literal token feature and a broader "royal title" feature, whereas standard SAEs merged them into a single redundant representation.

Downstream Performance and Efficiency

OrtSAE achieved a +6% improvement on the Spurious Correlation Removal (SCR) task, indicating that more atomic features lead to better intervention capabilities. While there is a marginal trade-off in "explained variance" (reconstruction accuracy), this is expected as the model is prevented from using redundant, overlapping features to minimize error. Crucially, OrtSAE remains highly efficient, with training times only ~1.11x the baseline, making it more practical than alternatives like Matryoshka SAEs.

Conclusion

OrtSAE provides a scalable and effective solution to the problems of feature absorption and composition. By leveraging geometric orthogonality as a proxy for conceptual atomicity, it uncovers a more diverse and interpretable set of features, providing a clearer lens into the internal workings of LLMs. Future work will explore using these atomic building blocks to map complex neural circuits in even larger models.

Summary updated

Abstract

Sparse autoencoders (SAEs) are a technique for sparse decomposition of neural network activations into human-interpretable features. However, current SAEs suffer from feature absorption, where specialized features capture instances of general features creating representation holes, and feature composition, where independent features merge into composite representations. In this work, we introduce Orthogonal SAE (OrtSAE), a novel approach aimed to mitigate these issues by enforcing orthogonality between the learned features. By implementing a new training procedure that penalizes high pairwise cosine similarity between SAE features, OrtSAE promotes the development of disentangled features while scaling linearly with the SAE size, avoiding significant computational overhead. We train OrtSAE across different models and layers and compare it with other methods. We find that OrtSAE discovers 9% more distinct features, reduces feature absorption (by 65%) and composition (by 15%), improves performance on spurious correlation removal (+6%), and achieves on-par performance for other downstream tasks compared to traditional SAEs.
BibTeX
@misc{korznikov2025ortsae,
  author = {Anton Korznikov and Andrey V. Galichin and Alexey Dontsov and Oleg Rogov and Elena Tutubalina and Ivan V. Oseledets},
  title = {{OrtSAE: Orthogonal Sparse Autoencoders Uncover Atomic Features}},
  year = {2025},
  eprint = {2509.22033},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.22033},
}
Figure