TL;DR
Can transformer activations be modeled simply as arbitrary combinations—a "bag"—of Sparse Auto-Encoder (SAE) features? By probing GPT-2 with activation perturbations, this work reveals that real activations possess strict internal geometric and statistical structure that naive combinations lack. While matching latent sparsity and directional cosine similarities makes synthetic activations behave like real ones in directional sensitivity tests, synthetic activations still fail to replicate the protective "activation plateaus" (noise-suppression zones) found around natural model activations.
What Problem Does This Paper Address?
In mechanistic interpretability, Sparse Auto-Encoders (SAEs) are popular tools for decomposing polysemantic residual stream vectors into interpretable, monosemantic feature directions. However, a fundamental question remains: Are internal representations truly just linear superpositions of independent SAE latents?
Previous research demonstrated that neural networks display two striking perturbation dynamics:
- Directional sensitivity: Perturbing an early-layer activation toward another real activation causes a sharp, step-function-like "blowup" in the final layer's representation much sooner than perturbing toward random Gaussian noise.
- Activation plateaus: Real activations are surrounded by a local stable basin (plateau) where small perturbations are suppressed—a hypothesized error-correction mechanism for superposition.
This paper tests whether synthetic activations constructed from combinations of SAE latents can faithfully recreate these dynamics, or whether real activations require specific, non-trivial latent geometry.
Methods
The authors perturb GPT-2 small residual activations at Layer 1 (blocks.1.hook_resid_pre) at the final token position along directions defined by different target vector types:
- Non-SAE Baselines:
- Model-Generated (Real): Activations sampled directly from runs on text.
- Random: Gaussian vectors matched to the empirical mean and covariance of real activations.
- Synthetic Compositions:
- Synthetic-Random: Randomly sampled SAE latents assigned the activation magnitudes of a base token.
- Synthetic-Baseline ("Bag of Latents"): Preserves latent sparsity and activation magnitudes by swapping active latents with similarly sparse latents.
- Synthetic-Structured: Explicitly incorporates geometry. It identifies the dominant active latent, constrains its sparsity and cosine similarity to base activations, and ensures remaining active latents match their empirical cosine similarities relative to the top latent.
The authors evaluate responses using Layer 11 L2 distance (and logit KL divergence) across perturbation steps, tracking the Maximum Slope (MS) step where blowups happen, and the Activation Plateau (AP) step where the network begins deviating from stability.
Datasets
- OpenWebText: Random 10-token prompts are sampled from OpenWebText to run inference through GPT-2 small and harvest 1,000 Layer 1 activations.
- No new training datasets or benchmarks were created; evaluations rely on standard pretrained GPT-2 small SAEs from
sae-lens.
Key Insights and Ideas
- Activations are not just "bags of latents": Randomly combining active SAE latents (even when matching individual sparsity and strength) yields perturbations that behave closer to random Gaussian noise than to real activations.
- Hierarchical latent geometry matters: Real activations are dominated by a single "top" latent that accounts for ~49% of the activation norm (out of ~21 active latents on average). Enforcing pairwise cosine similarity between other active latents and this top latent dramatically improves the fidelity of synthetic activations.
- Sensitivity transfers, but plateaus do not: While
synthetic-structuredvectors trigger early blowups matching real activations under directional sensitivity tests, they completely fail to produce stable activation plateaus around themselves. - SAE reconstruction error is not to blame: Target perturbations directed at direct SAE reconstructions ($\text{decode}(\text{encode}(A))$) behave almost identically to uncompressed activations ($KS = 0.02$). The shortcomings of synthetic activations stem from latent selection and correlation heuristics, not autoencoder fidelity.
Key Results
- Directional Sensitivity: Perturbing toward
synthetic-structuredactivations mimics perturbations toward real activations far better than the naive baseline, reducing the Kolmogorov-Smirnov (KS) distance to the real distribution from 0.28 (synthetic-baseline) down to 0.11 (synthetic-structured), compared to 0.45 for random directions. - Mean Blowup Step (MS): Real activations trigger a maximum slope blowup at step 41.11,
synthetic-structuredat 43.48,synthetic-baselineat 49.61, and random noise at 52.49. - Plateau Discrepancy: Perturbations initiated from real activations demonstrate extended plateaus before departing baseline output. Neither
synthetic-baselinenorsynthetic-structuredvectors exhibit this plateau, behaving instead similarly to random base noise.
Limitations and Future Work
- Scope: The empirical study is limited to Layer 1 of GPT-2 small on short context windows (10 tokens) and the final token position.
- Latent Co-occurrence: The geometric heuristic only accounts for cosine similarity to the primary latent; it ignores higher-order latent co-occurrence statistics.
- Missing Error-Correction Basins: Future work needs to explore what specific structural or non-linear properties give natural activations their error-correcting basins (plateaus) that synthetic mixtures currently fail to capture.
Summary updated