Evaluating Open-Source Sparse Autoencoders on Disentangling Factual Knowledge in GPT-2 Small

Maheep Chaudhary, Atticus GeigerarXiv 2024 · OpenReview

Abstract

A popular new method in mechanistic interpretability is to train high-dimensional sparse autoencoders (SAEs) on neuron activations and use SAE features as the atomic units of analysis. However, the body of evidence on whether SAE feature spaces are useful for causal analysis is underdeveloped. In this work, we use the RAVEL benchmark to evaluate whether SAEs trained on hidden representations of GPT-2 small have sets of features that separately mediate knowledge of which city is in which country and which continent. We evaluate four open-source SAEs for GPT-2 small against each other, with neurons serving as a baseline and linear features learned via distributed alignment search (DAS) serving as a skyline. For each, we learn a binary mask to select features that will be patched to change the country of a city without changing the continent, or vice versa. Our results show that SAEs struggle to reach the neuron baseline, and none come close to the DAS skyline.

Introduction

In the field of mechanistic interpretability, researchers aim to identify the "meaning-bearing components" of neural networks. While individual neurons are the natural candidates, they are often polysemantic—activating for a variety of unrelated concepts—making them difficult to use for precise causal analysis. Sparse Autoencoders (SAEs) have emerged as a potential solution, designed to "unpack" these messy activations into a higher-dimensional, sparse space of interpretable features.

While SAEs have shown promise in terms of low reconstruction error and human-interpretable features, their utility for causal analysis remains an open question. This study investigates whether SAE features can act as a "control panel" for precisely editing specific facts—such as a city's country—without affecting related knowledge, like the city's continent.

Methodology

The study utilizes the RAVEL benchmark, focusing on the disentanglement of entity attributes. Specifically, the task involves distinguishing between the country a city is in and the continent it is in. Due to the limitations of GPT-2 Small, the dataset was filtered to 40 single-token cities that the model reliably "knows."

Intervention Framework

The researchers employed Interchange Interventions to test causal control. For instance, if "country" features are identified, their values for "Paris" are swapped with those for "Tokyo." A successful intervention would result in the model identifying Paris as being in Japan while correctly maintaining that it is in Europe. To identify the relevant features in each space (neurons or SAE dimensions), Differential Binary Masking (DBM) was used to learn a binary mask for "country" vs. "continent" attributes.

Baselines and Comparisons

Four open-source SAEs (OpenAI, Apollo, and Bloom) were evaluated against:

  1. Neuron Baseline: Using raw neurons as the units of analysis.
  2. DAS Skyline: Distributed Alignment Search (DAS), a supervised method that finds the optimal linear subspace for a concept, representing the theoretical ceiling for linear feature performance.

Results and Analysis

The evaluation revealed significant challenges for current SAE architectures in matching even the basic neuron baseline for causal disentanglement.

Disentangle scores across layers for country and continent interventions.

The Reconstruction Tax and Knowledge Degradation

A critical finding was the "reconstruction tax": simply passing activations through an SAE often degrades the model's performance. In early layers (e.g., Layer 1), some SAEs like Apollo and Bloom caused a drop in base knowledge of over 50%. In extreme cases, the reconstruction process almost entirely destroyed the model's factual knowledge, as seen in the low "Reconstructed Knowledge" scores for certain architectures.

Disentanglement Performance

The DAS skyline achieved near-perfect disentanglement scores (~94-96%), confirming that the information is linearly separable within the model. However, most SAEs failed to outperform the neuron baseline.

Detailed performance metrics for various SAE architectures and baselines, highlighting the gap between unsupervised SAEs and the DAS skyline.

As shown in the table above, the OpenAI SAE was the only model to match the neuron baseline (~70-73%). Other architectures, particularly Apollo and Bloom, performed significantly worse, often failing to reach a 60% disentangle score in early layers. This suggests that the method of architectural constraints (e.g., top-k vs. L1 regularization) significantly impacts the causal utility of the resulting features.

Conclusion

While SAEs are a promising tool for interpretability, this work demonstrates that they are not a "silver bullet" for causal analysis. Current open-source SAEs for GPT-2 Small struggle to provide better disentanglement than raw neurons and can introduce significant reconstruction artifacts. Future work should explore whether these findings hold for larger models and more diverse factual domains, as well as investigate newer SAE training objectives that might yield more causally robust features.

Summary updated

Abstract

A popular new method in mechanistic interpretability is to train high-dimensional sparse autoencoders (SAEs) on neuron activations and use SAE features as the atomic units of analysis. However, the body of evidence on whether SAE feature spaces are useful for causal analysis is underdeveloped. In this work, we use the RAVEL benchmark to evaluate whether SAEs trained on hidden representations of GPT-2 small have sets of features that separately mediate knowledge of which country a city is in and which continent it is in. We evaluate four open-source SAEs for GPT-2 small against each other, with neurons serving as a baseline, and linear features learned via distributed alignment search (DAS) serving as a skyline. For each, we learn a binary mask to select features that will be patched to change the country of a city without changing the continent, or vice versa. Our results show that SAEs struggle to reach the neuron baseline, and none come close to the DAS skyline. We release code here: https://github.com/MaheepChaudhary/SAE-Ravel
BibTeX
@misc{chaudhary2024evaluating,
  author = {Maheep Chaudhary and Atticus Geiger},
  title = {{Evaluating Open-Source Sparse Autoencoders on Disentangling Factual Knowledge in GPT-2 Small}},
  year = {2024},
  eprint = {2409.04478},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.04478},
}
Figure