Abstract
Sparse Autoencoders (SAEs) are a primary tool for mechanistic interpretability, aiming to decompose Large Language Model (LLM) activations into monosemantic latents. However, narrow SAEs often fail to achieve this, instead producing polysemantic mixtures. This paper identifies "feature hedging" as a fundamental failure mode where narrow models merge correlated features to minimize reconstruction error. We introduce the "hedging degree" metric and Balanced Matryoshka SAEs to mitigate this effect, providing a path toward more interpretable narrow models.
Introduction
Mechanistic interpretability seeks to understand the internal logic of LLMs by mapping dense activation vectors to sparse, human-interpretable "latents." Sparse Autoencoders (SAEs) are the standard architecture for this task. Ideally, an SAE achieves monosemanticity: a one-to-one mapping where each latent corresponds to a single conceptual feature, such as "Python syntax" or "French geography."
While wide SAEs often achieve high interpretability, smaller, narrower models frequently struggle. This research identifies a specific phenomenon called feature hedging that explains why narrow SAEs "hedge their bets" by blurring distinct but correlated features into single, messy latents.
The Problem: Hedging vs. Absorption
Previous research has highlighted feature absorption, where a general "parent" latent in a wide SAE swallows more specific "child" features. Feature hedging is the opposite failure mode occurring at the narrow end of the spectrum. When an SAE lacks the capacity (width) to represent every feature, it is mathematically incentivized by the Mean Squared Error (MSE) loss to create polysemantic mixtures.
In the case of independent features, an SAE with sufficient width can easily map latents to features 1:1, achieving the desired monosemanticity.
Methodology
The authors investigated hedging through three primary lenses:
- Synthetic Toy Models: By creating datasets with controlled correlations (independent, hierarchical, positively correlated, and anti-correlated), they demonstrated that MSE reconstruction loss inherently favors hedging in narrow bottlenecks. The model "prefers" a noisy sum of two correlated features over a perfect representation of only one, as the former reduces total error across more samples.
- Quantifying Hedging: A new metric, the Hedging Degree ($h$), was developed. It measures how much information can be "pulled out" of existing latents by adding a small number of new latents to a pre-trained model. A high $h$ indicates that existing latents were hiding components of other features.
- Balanced Matryoshka SAEs: Matryoshka SAEs nest smaller models within larger ones. To prevent the inner (narrower) levels from suffering excessive hedging, the authors introduced a tunable loss-scaling factor ($\beta_m$). This allows for balancing the pressure between feature merging (hedging) in narrow levels and feature suppression (absorption) in wider levels.
Key Insights
- Width is Not Neutral: Reducing SAE width doesn't just result in missing features; it actively distorts the features that are captured. Narrower SAEs are incentivized to create polysemantic mixtures to minimize loss.
- MSE as a Driver: While $L_1$ sparsity penalties are often blamed for interpretability issues, the standard MSE loss is the primary driver of hedging. The model effectively "bets" on multiple features simultaneously.
- Correlation Sign Matters: Positive correlation leads to additive hedging, while anti-correlation can lead to "negative hedging," where a latent represents one concept while actively suppressing another.
Results and Evaluation
Experiments on Gemma-2-2b and Llama-3.2-1b using the Pile corpus confirmed that hedging is pervasive in real-world LLMs, especially at widths below 4096 latents. Qualitative case studies showed that adding latents for specific features (like CSS syntax) clarified existing, previously "hedged" latents (like HTML attributes).
Using SAEBench, the authors found that a scaling coefficient of $\beta \approx 0.75$ in Balanced Matryoshka SAEs provided the best trade-off, outperforming standard architectures on Targeted Probe Perturbation (TPP) and Spurious Concept Removal (SCR) benchmarks. This balancing act prevents the inner levels from becoming overly polysemantic due to hedging.
Conclusion
Feature hedging represents a significant hurdle for deploying efficient, narrow SAEs. By understanding that MSE loss drives this behavior, researchers can better design architectures like Balanced Matryoshka SAEs that maintain interpretability across varying widths. Future work may involve dynamic, per-latent balancing to further refine the monosemanticity of discovered features.
Summary updated