TL;DR
This paper introduces $k^*$, a metric for "readout sparsity" that measures the minimum number of features needed to extract task-specific information from a language model (LM). The researchers discovered that while different LMs vary in scale and training, they organize information in remarkably similar ways, and that supervised training actively "concentrates" task information into sparser, more accessible representations.
What Problem Does This Paper Address?
Researchers often use "linear probes"—simple classifiers trained on a model's internal activations—to see if a model "knows" a certain task (like identifying sentiment or detecting harmful content). However, a standard dense probe uses the entire representation (thousands of dimensions), which doesn't tell us how that information is actually organized.
Is the information spread thinly across the whole model, or is it concentrated in a few key "features"? This distinction matters because it influences how easily a model can be fine-tuned, how robust it is to pruning, and how it might "forget" tasks during continual learning. This paper moves beyond asking if information is present to asking how sparsely it is stored.
Methods
The authors define $k^*$ as the smallest number of features required to recover 95% of the performance of a "dense" probe (one that uses all available features).
To find $k^*$, they use a fixed search protocol involving:
- Feature Ranking: Identifying the most relevant features using statistical t-scores.
- L1 Regularization: Using "Lasso" style regression to identify the most efficient subsets of features.
- Cross-Model Matching: Using Sparse Autoencoders (SAEs) to see if the specific features identified in one model (e.g., a "Japanese language" feature) correspond to similar features in a model from a different lab.
- Controlled Synthetic Experiments: Training small models on purely synthetic tasks to isolate exactly how supervision changes the sparsity of a representation.
Datasets
The study evaluates 19 different LMs (including the Gemma, Llama, Mistral, OLMo, and Qwen families) across 14 core tasks. These tasks include:
- Dispatch tasks: Language identification, tool selection, and intent classification (e.g., CLINC150, Banking77).
- Safety/Judgment tasks: Harmfulness detection (WildGuardMix), jailbreak detection, and fact-checking (FEVER).
- Reasoning/Correctness: Math and code bug detection (ProcessBench, Devign).
Key Insights and Ideas
- Universal Sparsity Patterns: Despite being trained by different organizations on different data, base models show a striking agreement on which tasks are "narrow" (require few features) and which are "wide" (require many). For example, Language ID always needs ~1 feature, while Harmfulness detection needs hundreds.
- Supervision as a Compressor: The most "aha" moment comes from synthetic experiments. When a model is supervised on a specific target, it organizes that information into a single feature, even if the logic is complex. Readable information that wasn't explicitly supervised remains "wide," requiring tens of features to extract.
- Instruction Tuning's Impact: Instruction tuning (IT) selectively reduces $k^*$ for tasks relevant to the tuning (like safety and fact-checking). This suggests that the "chat" version of a model isn't just better at following instructions; it has physically reorganized its internal map to make safety-relevant information more accessible.
Key Results
- Predicting Learning Requirements: The authors found an empirical scaling law: $N^* \propto (k^*)^{0.65}$. This means the sparsity of a task in a "teacher" model can predict how many labeled examples ($N^*$) a "student" model will need to learn that same task.
- Behavioral Forgetting vs. Linear Readability: In continual learning, a model might "forget" how to perform an old task (the classification head fails). However, the internal $k^*$ measurements show that the information is often still there and just as readable as before; it's the "head," not the "knowledge," that broke.
- Pruning Vulnerability: "Wide" tasks (high $k^*$) are significantly more vulnerable to weight pruning. If a task's information is spread across many features, losing a random 60% of weights is much more likely to degrade performance than if the information were concentrated in a single feature.
Limitations and Future Work
- Upper Bound Nature: The $k^*$ metric is an upper bound based on a specific search protocol; it's possible even smaller subsets exist that the protocol missed.
- The "Why" of Intermediate Sparsity: While synthetic targets become 1-feature readouts, many real-world tasks in pretrained models have $k^*$ values between 10 and 100. The mechanism that produces these intermediate levels of concentration is still unknown.
- Intervention Challenges: While $k^*$ is great for diagnosing model changes, the authors' attempts to use it for interventions (like protecting specific weights during pruning) did not yet outperform simple random baselines. This suggests a gap between knowing which features are used and knowing which specific weights sustain them.
Summary updated