Evaluating Synthetic Activations composed of SAE Latents in GPT-2

Giorgi Giglemiani, Nora Petrova, Chatrik Singh Mangat, Jett Janiak, Stefan Heimersheim

Introduction

In mechanistic interpretability, Sparse Auto-Encoders (SAEs) are widely regarded as the "gold standard" for decomposing the complex activations of Large Language Models (LLMs) into monosemantic, interpretable features. However, a critical question persists: is a model activation merely a "bag of SAE features," or does it possess a deeper internal structure that current SAEs fail to capture? This study investigates whether synthetic activations—constructed by combining SAE latents—can replicate the emergent behaviors of real model data, specifically "activation plateaus" and "directional sensitivity."

Experimental Framework

The researchers utilized GPT-2 small as a testbed, performing perturbation experiments on activations from Layer 11 using prompts from the OpenWebText dataset. By starting with a base activation and incrementally nudging it toward a target direction, they measured the rate of change in the model's final-layer output using L2 distance and KL divergence.

To evaluate the "bag of features" hypothesis, three types of synthetic target activations were developed:

  1. Synthetic-random: Combinations of randomly selected SAE latents.
  2. Synthetic-baseline: Latents selected to match the sparsity (number of active features) of real activations.
  3. Synthetic-structured: Latents selected to match both the sparsity and the geometric relationships (cosine similarity) of real feature sets.

The "Blowup" Phenomenon and Directional Sensitivity

A key characteristic of real model activations is their directional sensitivity. When an internal state is perturbed toward another "real" activation, the output changes abruptly—a phenomenon termed a "blowup." This occurs much faster than perturbations toward random directions.

L2 distance and KL divergence show that model-generated directions cause an earlier and sharper "blowup" in output compared to random directions.

As illustrated in Figure 2, the model is significantly more sensitive to perturbations in meaningful directions. This suggests that the "manifold" of real activations is distinct and that the model is tuned to respond to specific feature combinations rather than arbitrary directions in activation space.

Evaluating Synthetic Constructions

The study found that simply matching sparsity is insufficient; the "Synthetic-baseline" activations behaved similarly to random noise. However, when the geometric relationships (angles) between features were replicated in the "Synthetic-structured" approach, the activations began to mimic real data more closely.

Distributions and CDFs of the Maximum Slope (MS) metric show that Synthetic-structured activations (green) more closely approximate real model behavior (orange) than baseline methods.

Quantitatively, real activations reached a "blowup" point (Maximum Slope) at an average perturbation step of 41.11, while random directions lagged at 52.49. The Synthetic-structured method achieved a step of 43.48, significantly outperforming the Synthetic-baseline (49.61). Statistical similarity, measured via the Kolmogorov-Smirnov (KS) statistic, further confirmed this: the Synthetic-structured method scored 0.11, whereas random activations scored 0.45 (where lower is better).

The Missing Error-Correction Plateau

Despite the success of geometric matching in replicating sensitivity, a major discrepancy remains: the "activation plateau." Real model activations exhibit a "safe zone" (AP step ~15-20) where small amounts of noise do not significantly alter the output. This is likely an error-correction mechanism designed to handle interference in feature superposition.

None of the synthetic methods, including the structured approach, were able to recover this plateau. Synthetic activations showed almost no noise tolerance (AP step ~5), behaving like random noise in this regard. This suggests that SAEs are not yet capturing the mechanisms the model uses to suppress noise or ignore low-magnitude feature interference.

Conclusion

While matching the geometry of SAE features significantly improves the "realism" of synthetic activations, it does not fully bridge the gap. The absence of activation plateaus in synthetic data indicates that there is a functional layer of neural computation—possibly related to error correction and noise suppression—that is not yet captured by the "bag of latents" model. Future work will focus on feature co-occurrence statistics and discovering the specific activation thresholds that define the model's robustness.

Summary updated

Abstract

Sparse Auto-Encoders (SAEs) are commonly employed in mechanistic interpretability to decompose the residual stream into monosemantic SAE latents. Recent work demonstrates that perturbing a model's activations at an early layer results in a step-function-like change in the model's final layer activations. Furthermore, the model's sensitivity to this perturbation differs between model-generated (real) activations and random activations. In our study, we assess model sensitivity in order to compare real activations to synthetic activations composed of SAE latents. Our findings indicate that synthetic activations closely resemble real activations when we control for the sparsity and cosine similarity of the constituent SAE latents. This suggests that real activations cannot be explained by a simple "bag of SAE latents" lacking internal structure, and instead suggests that SAE latents possess significant geometric and statistical properties. Notably, we observe that our synthetic activations exhibit less pronounced activation plateaus compared to those typically surrounding real activations.
BibTeX
@misc{giglemiani2024evaluating,
  author = {Giorgi Giglemiani and Nora Petrova and Chatrik Singh Mangat and Jett Janiak and Stefan Heimersheim},
  title = {{Evaluating Synthetic Activations composed of SAE Latents in GPT-2}},
  year = {2024},
  eprint = {2409.15019},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.15019},
}
Figure