Probing by Analogy: Decomposing Probes into Activations for Better Interpretability and Inter-Model Generalization

Patrick Leask, Noura Al MoubayedMechInterp @ NeurIPS 2025 · OpenReview

TL;DR

This paper introduces "analogous probing," a new method to decompose linear probes into a sparse set of model activations. This approach offers better interpretability and significantly improves how well probes generalize across different language models and training checkpoints, addressing key limitations of traditional linear probes.

What Problem Does This Paper Address?

Interpreting the inner workings of large language models (LLMs) is crucial for understanding their behavior, ensuring safety, and improving their capabilities. A common technique for this is "probing," where researchers train simple linear classifiers (linear probes) on the frozen activations of an LLM to see if specific concepts (like truthfulness or sentiment) can be detected. While effective, linear probes have several drawbacks: they can latch onto spurious correlations in the training data, overfit in high-dimensional spaces, and their performance often degrades significantly when transferred to different models or checkpoints. This makes it hard to reliably understand how LLM representations evolve and generalize. The paper aims to develop a more robust and interpretable probing method that overcomes these limitations.

Methods

The core contribution is analogous probing, a method inspired by sparse autoencoder (SAE) decomposition techniques. Instead of training a linear probe directly on activations, analogous probing decomposes the probe's weight vector into a weighted sum of a small number of activations from the training data. This is framed as a sparse coding problem, solved using orthogonal matching pursuit. The key idea is that these "analogous" probes are defined by their relationship to specific examples (activations) rather than absolute directions in the activation space. This makes them invariant to affine transformations (like scaling or rotation) of the representation space, a property that traditional linear probes lack.

Datasets

The paper uses six datasets from Marks and Tegmark [2023] for evaluating truthfulness. These datasets are described as "small" and probe for a single target variable (truth). The models used include families from Gemma, Llama, and Mistral, as well as various checkpoints of the LLM360 Amber model.

Key Insights and Ideas

  • Probes can be decomposed into examples: The core insight is that a linear probe's direction can be represented as a sparse combination of activations from the training data, offering a more interpretable view than a single, abstract vector.
  • Affine invariance is key for generalization: The paper argues that the poor generalization of linear probes across models and checkpoints is partly due to their sensitivity to affine transformations of the representation space. Analogous probes, by being invariant to these transformations, show much better transferability.
  • Representation space changes are subtler than thought: The improved generalization of analogous probes suggests that the underlying structure of LLM representations might change less drastically during fine-tuning or across checkpoints than previously indicated by the performance drop of linear probes.

Key Results

  • Comparable performance to linear probes: Analogous probes achieve performance comparable to trained linear probes, even with a very small number of constituent activations (as few as 10).
  • Superior to mass-mean probes: Analogous probes significantly outperform mass-mean probes, especially in reconstruction performance.
  • Improved inter-model generalization: Analogous probes show much better performance when transferred between different models and checkpoints compared to naive linear probes. For instance, on Llama models, analogous probes performed significantly better than other methods when transferring.
  • Robustness to affine transformations: Empirically, analogous probes maintained high accuracy (98.5%) after random affine transformations of activations, while linear probes dropped to 57.7%.

Limitations and Future Work

  • Limited dataset diversity: The study used only six datasets, merging positive and negative variants. Future work will include more datasets.
  • Interpretability evaluation: The interpretability of the probe decompositions themselves on downstream tasks was not evaluated. Future work plans to use selectivity experiments for this.
  • Comparison to other methods: The paper notes a lack of literature for strong baselines and suggests comparing against whitened mass-mean and contrast-consistent search probes.
  • Model and layer limitations: The experiments were conducted on smaller models and a single layer. Further ablations across different model sizes and layers are needed.
  • Understanding Llama performance: The paper wonders why analogous probes outperform other methods so significantly on Llama models and plans to investigate this further.

Summary updated

Abstract

Linear probes have been used to demonstrate that LLM activations linearly encode high-level properties of the input, such as truthfulness, and that these directions can evolve significantly during fine-tuning and training. However, despite their seeming simplicity, linear probes can have complex geometric interpretations, leverage spurious correlations, and lack selectivity. We present a method for decomposing linear probe directions into weighted sums of as few as 10 model activations, whilst maintaining task performance. These probes are also invariant to affine transformations of the representation space, and we demonstrate that, in some cases, poor base to fine-tune probe generalization performance is partially due to simple transformations of representation subspaces, and the structure of the representation space changes less than indicated by other methods.
BibTeX
@inproceedings{leask2025probing,
  author = {Patrick Leask and Noura Al Moubayed},
  title = {{Probing by Analogy: Decomposing Probes into Activations for Better Interpretability and Inter-Model Generalization}},
  booktitle = {MechInterp @ NeurIPS},
  year = {2025},
  url = {https://openreview.net/forum?id=IhI1qGyk6S},
}
Figure