TL;DR
This paper introduces SFAL, a novel and efficient method for evaluating how well auto-generated explanations (auto-interpretations) of sparse autoencoder (SAE) features align with the features' actual behavior in large language models (LLMs). SFAL offers a scalable and cost-effective alternative to current LLM-based evaluation methods, demonstrating strong correlation with human judgments.
What Problem Does This Paper Address?
Interpreting the internal workings of large language models (LLMs) is crucial for their safe and reliable deployment. Sparse Autoencoders (SAEs) are a promising technique for this, decomposing complex LLM activations into simpler, more interpretable "features." Auto-interpretability methods aim to generate human-readable explanations for these features. However, evaluating the quality of these explanations is challenging. Current methods often rely on prompting other LLMs to score the explanations against feature activations. This approach is computationally expensive, prone to noise, and doesn't scale well to the millions of features found in large SAEs. The paper addresses this gap by proposing a more efficient and scalable evaluation metric that leverages the inherent structure of the SAE feature space itself.
Methods
The core of the proposed method, SFAL (Semantic-Functional Alignment Scores), is to quantify the alignment between two aspects of an SAE feature: its semantic neighborhood and its functional neighborhood.
- Semantic Representation: Each SAE feature's auto-interpretation (a textual description) is encoded into a vector using a language model. The semantic neighborhood of a feature is defined as the set of other features whose auto-interpretation vectors are most similar (using cosine similarity).
- Functional Representation: A feature's functional behavior is captured by its co-occurrence statistics with other features. This is represented by a co-occurrence matrix, from which the phi coefficient is calculated to measure the strength of association between feature pairs. The functional neighborhood of a feature is the set of other features with the highest positive phi coefficient values.
- SFAL Score: The alignment between the semantic and functional neighborhoods is then calculated using Normalized Discounted Cumulative Gain (NDCG). A high SFAL score indicates that features semantically similar to a given feature are also functionally associated with it, suggesting a consistent and reliable interpretation.
Datasets
The paper evaluates their method on SAEs trained on two large language models:
- Gemma-2-9b
- Llama-3.1-8b
The co-occurrence matrices were built by processing 50,000 documents from the respective SAE training datasets.
Key Insights and Ideas
- Bridging Semantic and Functional Gaps: The paper highlights that purely semantic evaluations of auto-interpretations might miss crucial functional aspects. SFAL bridges this by enforcing consistency between what an explanation says a feature does (semantic) and how it actually behaves with other features (functional).
- Efficiency is Key: The primary motivation is to move away from expensive LLM-based scoring. SFAL achieves this by leveraging matrix computations and similarity metrics, making it significantly faster and cheaper.
- Scalability for Real-World Deployment: The proposed method is designed to scale to millions of features, a critical requirement for practical applications of interpretability in large models.
- Human Correlation: A key validation is the strong correlation between SFAL scores and human judgments, suggesting that the metric captures meaningful aspects of interpretability.
Key Results
- SFAL demonstrates a significant computational efficiency improvement over LLM-based evaluation pipelines, requiring only 2 GPU hours on a single NVIDIA A100 for experiments.
- The study found that SFAL scores generally assign lower values than fuzzing scores, indicating a more selective and potentially less optimistic evaluation approach.
- SFAL showed a strong positive correlation with human judgments (soundness and completeness) across multiple embedding models tested on the Gemma-2-9b SAEs. While slightly underperforming fuzzing on Llama-3.1-8b in one instance, it remained consistently significant.
- The human evaluation, involving four expert users, showed substantial inter-rater agreement, lending credibility to the human judgments used for validation.
Limitations and Future Work
- Co-occurrence as a Proxy: The authors acknowledge that co-occurrence is an imperfect proxy for true functional linkage. Cases where features are functionally correlated but semantically dissimilar (e.g., transitive predicates and their direct objects) can lead to noise.
- Fixed-K Neighborhoods: The current implementation uses fixed-size neighborhoods (top-K). The authors suggest that adaptive strategies based on feature sparsity could be explored for better adaptation.
- Limited Human Evaluation Pool: The human evaluation involved a small pool of expert users. Future work should involve larger and more diverse user studies.
- Generalization: Future work will explore more expressive similarity metrics and investigate how SFAL generalizes across different LLM architectures and domains.
Summary updated