TL;DR
Interpretability tools—like vectors that trigger model refusal or probes that detect truthfulness—are not unique to specific models. This paper demonstrates that these tools can be transferred across different LLM families (e.g., Llama to Qwen) using simple linear maps fit on unlabeled data, achieving near-perfect recovery of the tool's original effect.
What Problem Does This Paper Address?
In the field of mechanistic interpretability, researchers often find "directions" in a model's internal activation space that correspond to specific behaviors or concepts. For example, you might find a "refusal vector" that, when added to a model's activations, forces it to decline harmful requests.
The problem is that these tools are currently "bespoke." If you find a refusal direction in Llama-3, you generally have to start from scratch to find the equivalent in Gemma-2. This is labor-intensive and requires labeled data for every new model. It also leaves open a fundamental question: are these models learning the same internal "concepts," or is every model's internal world completely different?
Methods
The authors propose a "carry-over" approach. Instead of rebuilding a tool for a target model, they learn a linear map (specifically, ridge regression) between the residual streams of a source model and a target model.
Key aspects of the methodology include:
- Unlabeled Mapping: The map is fit using the activations of both models on the same set of unlabeled prompts. No target-side labels (e.g., "this is a harmful prompt") are needed.
- Linearity: Despite the non-linear nature of LLMs, the authors find that the correspondence between different models' residual streams is remarkably linear.
- Coverage Score: They introduce a label-free "coverage score" ($c$) that predicts how well a tool will transfer based on whether the mapping data spans the direction the tool operates in.
- Causal Testing: They don't just look at correlations; they test the transferred refusal direction by adding it to (steering) or ablating it from (removing) the target model's activations during text generation.
Datasets
The study evaluates transfer across 110 pairs of models (from 1B to 72B parameters) using several datasets:
- General Mapping Data: 5,400 ordinary questions across nine domains (e.g., GSM8K, ARC, TruthfulQA).
- Task-Specific Prompts: Unlabeled harmful/harmless instructions (from Alpaca, LLM-LAT, etc.), answer candidates for ARC-Challenge and TruthfulQA, and SST-2 movie reviews.
- Evaluation Benchmarks: ARC-Challenge (correctness), TruthfulQA (truthfulness), and SST-2 (sentiment).
Key Insights and Ideas
- "Coverage" is the Secret Sauce: A map fit only on general text might only recover ~54% of a refusal tool's effect. However, adding just 16 unlabeled harmful prompts to the mapping data raises recovery to 102%. This explains why previous attempts at cross-model mapping often failed: the maps didn't "see" the specific regions of activation space where the tools lived.
- Universal Mechanisms: The fact that a refusal direction from a 1B parameter Llama model can perfectly steer a 70B parameter Llama model (or even a Qwen model) suggests that these models are converging on similar internal mechanisms for high-level concepts.
- Outperforming Native Unsupervised Tools: In many cases, a tool "carried" from another model outperforms tools built directly in the target model using unsupervised methods like Principal Component Analysis (PCA). Carrying a tool effectively "imports" the supervision from the source model.
Key Results
The authors measured "recovery"—how much of the target model's own native tool effect was reproduced by the carried tool.
- Refusal Steering: With proper coverage, the carried tool recovered 102% of the native refusal effect on average.
- Correctness & Sentiment: Carried read-out tools recovered 88% to 98% of the native performance.
- Cross-Family Transfer: Transfer worked consistently across different families (Llama, Qwen, Gemma, Phi, Mistral) and between base and instruction-tuned versions of the same model.
- Specificity: Control directions (random or orthogonal vectors) recovered at most 1%, proving the transfer is highly specific to the intended concept.
Limitations and Future Work
- White-Box Requirement: This method requires access to internal activations (residual streams), meaning it cannot be used on "black-box" API-only models like GPT-4o or Claude 3.5.
- Linearity Assumption: The study focused on linear directions and probes. It remains to be seen if more complex interpretability structures, like sparse autoencoder (SAE) features or multi-step circuits, transfer as easily.
- The "Why" of Pairwise Similarity: While the coverage score predicts if a tool will transfer for a given map, it doesn't fully explain why some model pairs are inherently more "mappable" than others (measured by KL divergence in the paper). Exploring the root causes of this representational alignment is a promising future direction.
Summary updated