Separating Interpretability and Reconstruction in Sparse Autoencoders

Anh Tien Nguyen, Dimitris Samaras, Mahdi S. HosseiniarXiv 2026 · arXiv

TL;DR

DualSAE decouples the conflicting goals of reconstruction fidelity and interpretability in Sparse Autoencoders (SAEs) by introducing a dedicated "Residual Branch" to handle non-semantic information. This architectural split, enforced by a novel decorrelation loss, significantly boosts both the accuracy of reconstructions (~29% improvement) and the semantic clarity of learned features (~34% improvement) across both vision and language models.

What Problem Does This Paper Address?

Mechanistic interpretability aims to "reverse engineer" neural networks by decomposing their dense activations into sparse, human-understandable features. Sparse Autoencoders (SAEs) are the primary tool for this, but they face a fundamental tension: a single sparse representation is forced to serve two masters.

  1. Reconstruction: The model must perfectly recreate the original input.
  2. Interpretability: The features should represent clean, coherent concepts (e.g., "dog faces" or "temporal expressions").

The problem is that perfect reconstruction requires capturing everything, including non-semantic noise and low-level details that don't correspond to interpretable concepts. In standard SAEs, this "non-semantic" information gets entangled with semantic features, muddying their meaning and creating a trade-off where better reconstruction often leads to worse interpretability.

Standard SAE vs. DualSAE comparison. Standard SAEs entangle semantics with reconstruction in one code, while DualSAE separates them.

Methods

The authors propose DualSAE, a modular framework that splits the representational responsibility into two branches:

  • The Sparse Branch: A standard SAE architecture responsible for capturing the "dominant semantics" of the input in a sparse latent code.
  • The Residual Branch: A plug-in module (typically a simple projector or MLP) designed to model the remaining non-semantic information omitted by the Sparse Branch.

To ensure these branches don't just learn the same thing, the authors introduce two key training strategies:

  1. Decorrelation Loss: This loss function explicitly penalizes the Residual Branch if its outputs align with the active directions in the Sparse Branch's decoder. This forces the Residual Branch to focus only on information that the Sparse Branch cannot represent.
  2. Two-Stage Training: The model first "warms up" by training only the Sparse Branch as a standard SAE. Once the semantic features have stabilized, the Residual Branch is activated, and the two are trained jointly.
The DualSAE architecture, showing the Sparse Branch capturing semantic features and the Residual Branch capturing the non-semantic residual.

Datasets

The authors evaluated DualSAE across diverse configurations:

  • Vision: Features extracted from ImageNet-1K and OpenImagesV7 using three different encoders: CLIP, SigLIP2, and ViT-L/16.
  • Language: Activations from the Pythia-410m model trained on OpenWebText (web text) and GitHub code (programming languages).

Key Insights and Ideas

  • The "Residual is Noise" Hypothesis: The authors demonstrate that in standard SAEs, the reconstruction error (the residual) contains very little semantic information. By measuring "class purity," they show that while the reconstructed part of an image embedding is highly predictive of its class, the residual part is essentially non-semantic.
  • Specialization through Separation: By giving the non-semantic information a "place to go" (the Residual Branch), the Sparse Branch is freed from the burden of perfect reconstruction. This allows its latent features to specialize and become more "monosemantic"—meaning each feature responds to one, and only one, clear concept.
  • Joint Optimization is Key: Simply adding a residual branch to a frozen, pre-trained SAE helps reconstruction but does not improve interpretability. The Sparse Branch needs to be trained alongside the Residual Branch to "learn to let go" of the non-semantic details.
Figure 2. Left: Distribution of class purity for nearest neighbors retrieved using reconstruction ˆx and residual r from BatchTopK SAE. Retrievals from ˆx are substantially purer than those from r. Right: Example retrievals for the same query image using ˆx (top) and r (bottom). The reconstruction ˆx shows consistent semantics, while the residual is non-semantic.

Key Results

  • Quantitative Gains: DualSAE achieved a mean relative improvement of 29% in reconstruction fidelity ($R^2$) and 34% in interpretability (measured by MonoScore) compared to standard SAE baselines.
  • Qualitative Clarity: In language models, DualSAE features were more selective. For example, a "time-related" latent in DualSAE responded specifically to years and dates, whereas the standard SAE version also activated on unrelated tokens.
  • Downstream Utility: The cleaner features translated to better performance in practical applications:
  • Topic Modeling: Produced more diverse and concrete visual topics.
  • LLM Steering: Successfully shifted the sentiment of generated text more coherently than standard SAEs.

Limitations and Future Work

  • Hyperparameter Sensitivity: DualSAE introduces new hyperparameters, such as the decorrelation coefficient and the length of the warm-up stage, which may require tuning for different models.
  • Frozen Encoders: The current study uses frozen pre-trained encoders. Future work could explore end-to-end training where the encoder itself is updated.
  • Modality Expansion: While tested on vision and language, the framework's effectiveness on other modalities (like audio or multimodal reasoning) remains to be seen.

Summary updated

Abstract

Sparse Autoencoders (SAEs) are effective tools for mechanistic interpretability by decomposing neural representations into sparse and interpretable features. However, a single sparse representation in traditional SAEs is responsible for both reconstruction fidelity and interpretability, which are not guaranteed to align. To separate these objectives, we propose DualSAE, a modular framework that augments standard SAEs with a plug-in Residual Branch. A standard SAE serves as the Sparse Branch, learning a sparse code that contains the dominant semantics of the input. The added Residual Branch models the non-semantic information omitted by the Sparse Branch. We ensure that the contributions of the two branches towards the overall reconstruction have minimal overlap through a decorrelation loss. We also adopt a two-stage training strategy that first stabilizes sparse representation learning before coupling with the Residual Branch. Across diverse encoder-SAE configurations on two vision datasets and two language corpora, DualSAE achieves mean relative improvements of approximately 29% in reconstruction fidelity and 34% in interpretability (measured by monosemanticity score). Moreover, with more interpretable representations, DualSAE yields consistent performance gains in the downstream applications of topic modeling, hypothesis generation, and LLM steering. Qualitative analysis further suggests that the Sparse Branch indeed preserves dominant semantic content, whereas the Residual Branch captures non-semantic information to improve reconstruction fidelity.
BibTeX
@misc{nguyen2026separating,
  author = {Anh Tien Nguyen and Dimitris Samaras and Mahdi S. Hosseini},
  title = {{Separating Interpretability and Reconstruction in Sparse Autoencoders}},
  year = {2026},
  eprint = {2611.25165},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25165},
}
Figure