Introduction
In mechanistic interpretability, Sparse Auto-Encoders (SAEs) are widely regarded as the "gold standard" for decomposing the complex activations of Large Language Models (LLMs) into monosemantic, interpretable features. However, a critical question persists: is a model activation merely a "bag of SAE features," or does it possess a deeper internal structure that current SAEs fail to capture? This study investigates whether synthetic activations—constructed by combining SAE latents—can replicate the emergent behaviors of real model data, specifically "activation plateaus" and "directional sensitivity."
Experimental Framework
The researchers utilized GPT-2 small as a testbed, performing perturbation experiments on activations from Layer 11 using prompts from the OpenWebText dataset. By starting with a base activation and incrementally nudging it toward a target direction, they measured the rate of change in the model's final-layer output using L2 distance and KL divergence.
To evaluate the "bag of features" hypothesis, three types of synthetic target activations were developed:
- Synthetic-random: Combinations of randomly selected SAE latents.
- Synthetic-baseline: Latents selected to match the sparsity (number of active features) of real activations.
- Synthetic-structured: Latents selected to match both the sparsity and the geometric relationships (cosine similarity) of real feature sets.
The "Blowup" Phenomenon and Directional Sensitivity
A key characteristic of real model activations is their directional sensitivity. When an internal state is perturbed toward another "real" activation, the output changes abruptly—a phenomenon termed a "blowup." This occurs much faster than perturbations toward random directions.
As illustrated in Figure 2, the model is significantly more sensitive to perturbations in meaningful directions. This suggests that the "manifold" of real activations is distinct and that the model is tuned to respond to specific feature combinations rather than arbitrary directions in activation space.
Evaluating Synthetic Constructions
The study found that simply matching sparsity is insufficient; the "Synthetic-baseline" activations behaved similarly to random noise. However, when the geometric relationships (angles) between features were replicated in the "Synthetic-structured" approach, the activations began to mimic real data more closely.
Quantitatively, real activations reached a "blowup" point (Maximum Slope) at an average perturbation step of 41.11, while random directions lagged at 52.49. The Synthetic-structured method achieved a step of 43.48, significantly outperforming the Synthetic-baseline (49.61). Statistical similarity, measured via the Kolmogorov-Smirnov (KS) statistic, further confirmed this: the Synthetic-structured method scored 0.11, whereas random activations scored 0.45 (where lower is better).
The Missing Error-Correction Plateau
Despite the success of geometric matching in replicating sensitivity, a major discrepancy remains: the "activation plateau." Real model activations exhibit a "safe zone" (AP step ~15-20) where small amounts of noise do not significantly alter the output. This is likely an error-correction mechanism designed to handle interference in feature superposition.
None of the synthetic methods, including the structured approach, were able to recover this plateau. Synthetic activations showed almost no noise tolerance (AP step ~5), behaving like random noise in this regard. This suggests that SAEs are not yet capturing the mechanisms the model uses to suppress noise or ignore low-magnitude feature interference.
Conclusion
While matching the geometry of SAE features significantly improves the "realism" of synthetic activations, it does not fully bridge the gap. The absence of activation plateaus in synthetic data indicates that there is a functional layer of neural computation—possibly related to error correction and noise suppression—that is not yet captured by the "bag of latents" model. Future work will focus on feature co-occurrence statistics and discovering the specific activation thresholds that define the model's robustness.
Summary updated