TL;DR
This paper introduces "analogous probing," a new method to decompose linear probes into a sparse set of model activations. This approach offers better interpretability and significantly improves how well probes generalize across different language models and training checkpoints, addressing key limitations of traditional linear probes.
What Problem Does This Paper Address?
Interpreting the inner workings of large language models (LLMs) is crucial for understanding their behavior, ensuring safety, and improving their capabilities. A common technique for this is "probing," where researchers train simple linear classifiers (linear probes) on the frozen activations of an LLM to see if specific concepts (like truthfulness or sentiment) can be detected. While effective, linear probes have several drawbacks: they can latch onto spurious correlations in the training data, overfit in high-dimensional spaces, and their performance often degrades significantly when transferred to different models or checkpoints. This makes it hard to reliably understand how LLM representations evolve and generalize. The paper aims to develop a more robust and interpretable probing method that overcomes these limitations.
Methods
The core contribution is analogous probing, a method inspired by sparse autoencoder (SAE) decomposition techniques. Instead of training a linear probe directly on activations, analogous probing decomposes the probe's weight vector into a weighted sum of a small number of activations from the training data. This is framed as a sparse coding problem, solved using orthogonal matching pursuit. The key idea is that these "analogous" probes are defined by their relationship to specific examples (activations) rather than absolute directions in the activation space. This makes them invariant to affine transformations (like scaling or rotation) of the representation space, a property that traditional linear probes lack.
Datasets
The paper uses six datasets from Marks and Tegmark [2023] for evaluating truthfulness. These datasets are described as "small" and probe for a single target variable (truth). The models used include families from Gemma, Llama, and Mistral, as well as various checkpoints of the LLM360 Amber model.
Key Insights and Ideas
- Probes can be decomposed into examples: The core insight is that a linear probe's direction can be represented as a sparse combination of activations from the training data, offering a more interpretable view than a single, abstract vector.
- Affine invariance is key for generalization: The paper argues that the poor generalization of linear probes across models and checkpoints is partly due to their sensitivity to affine transformations of the representation space. Analogous probes, by being invariant to these transformations, show much better transferability.
- Representation space changes are subtler than thought: The improved generalization of analogous probes suggests that the underlying structure of LLM representations might change less drastically during fine-tuning or across checkpoints than previously indicated by the performance drop of linear probes.
Key Results
- Comparable performance to linear probes: Analogous probes achieve performance comparable to trained linear probes, even with a very small number of constituent activations (as few as 10).
- Superior to mass-mean probes: Analogous probes significantly outperform mass-mean probes, especially in reconstruction performance.
- Improved inter-model generalization: Analogous probes show much better performance when transferred between different models and checkpoints compared to naive linear probes. For instance, on Llama models, analogous probes performed significantly better than other methods when transferring.
- Robustness to affine transformations: Empirically, analogous probes maintained high accuracy (98.5%) after random affine transformations of activations, while linear probes dropped to 57.7%.
Limitations and Future Work
- Limited dataset diversity: The study used only six datasets, merging positive and negative variants. Future work will include more datasets.
- Interpretability evaluation: The interpretability of the probe decompositions themselves on downstream tasks was not evaluated. Future work plans to use selectivity experiments for this.
- Comparison to other methods: The paper notes a lack of literature for strong baselines and suggests comparing against whitened mass-mean and contrast-consistent search probes.
- Model and layer limitations: The experiments were conducted on smaller models and a single layer. Further ablations across different model sizes and layers are needed.
- Understanding Llama performance: The paper wonders why analogous probes outperform other methods so significantly on Llama models and plans to investigate this further.
Summary updated