Abstract
Sparse autoencoders (SAEs) are a critical technique for decomposing neural network activations into human-interpretable features. However, current SAEs often suffer from "identity crises": feature absorption, where general features are swallowed by specific ones, and feature composition, where independent concepts merge into a single representation. We introduce OrtSAE (Orthogonal SAE), a novel approach that mitigates these issues by enforcing geometric orthogonality between learned features. By implementing a computationally efficient training penalty, OrtSAE reduces feature absorption by 65% and composition by 15%, discovering 9% more unique, "atomic" features while maintaining high reconstruction fidelity.
The Challenge of Feature Identity
Mechanistic interpretability aims to identify the fundamental "atoms" of thought within Large Language Models (LLMs). While SAEs are the current standard for this task, they frequently fail in two ways:
- Feature Absorption: A broad concept (e.g., "words starting with E") is captured by multiple specific features (e.g., "elephant", "eagle"), leaving a "hole" in the representation of the general concept.
- Feature Composition: Distinct concepts (e.g., "red" and "square") are merged into a single composite feature ("red square"), hindering fine-grained analysis.
These failures occur because standard SAE objectives prioritize reconstruction and sparsity without explicitly discouraging feature overlap in the vector space.
Methodology: Enforcing Geometric Atomicity
OrtSAE addresses these issues by adding an orthogonality penalty to the standard SAE loss function. This penalty specifically targets and punishes high pairwise cosine similarity between the decoder feature vectors, forcing the model to find non-overlapping directions in the activation space.
To ensure this approach scales to massive models, the authors implemented two key optimizations:
- Chunk-wise Strategy: The latent space is randomly partitioned into smaller chunks, and the penalty is only computed within these chunks. This reduces computational complexity from quadratic to linear ($O(m)$).
- Sparse Computation: The orthogonality loss is calculated only every fifth training iteration, resulting in a negligible training overhead of approximately 4% compared to standard baselines.
Key Results and Insights
We evaluated OrtSAE on Gemma-2-2B and Llama-3-8B using the SAEBench suite. The results demonstrate that enforcing geometric orthogonality directly translates to conceptual "atomicity."
Reducing Failure Modes
As shown in the summary metrics, OrtSAE significantly outperforms traditional SAEs in reducing both absorption (-65%) and composition (-15%). By preventing features from "hiding" inside one another, OrtSAE discovered 9% more unique features compared to the BatchTopK baseline. Qualitative analysis supports this; for instance, OrtSAE successfully decomposed the token "Queen" into a literal token feature and a broader "royal title" feature, whereas standard SAEs merged them into a single redundant representation.
Downstream Performance and Efficiency
OrtSAE achieved a +6% improvement on the Spurious Correlation Removal (SCR) task, indicating that more atomic features lead to better intervention capabilities. While there is a marginal trade-off in "explained variance" (reconstruction accuracy), this is expected as the model is prevented from using redundant, overlapping features to minimize error. Crucially, OrtSAE remains highly efficient, with training times only ~1.11x the baseline, making it more practical than alternatives like Matryoshka SAEs.
Conclusion
OrtSAE provides a scalable and effective solution to the problems of feature absorption and composition. By leveraging geometric orthogonality as a proxy for conceptual atomicity, it uncovers a more diverse and interpretable set of features, providing a clearer lens into the internal workings of LLMs. Future work will explore using these atomic building blocks to map complex neural circuits in even larger models.
Summary updated