How Language Models Organize Task Information

Kaiyang Wan, Yuxia WangarXiv 2026 · arXiv

TL;DR

This paper introduces $k^*$, a metric for "readout sparsity" that measures the minimum number of features needed to extract task-specific information from a language model (LM). The researchers discovered that while different LMs vary in scale and training, they organize information in remarkably similar ways, and that supervised training actively "concentrates" task information into sparser, more accessible representations.

What Problem Does This Paper Address?

Researchers often use "linear probes"—simple classifiers trained on a model's internal activations—to see if a model "knows" a certain task (like identifying sentiment or detecting harmful content). However, a standard dense probe uses the entire representation (thousands of dimensions), which doesn't tell us how that information is actually organized.

Is the information spread thinly across the whole model, or is it concentrated in a few key "features"? This distinction matters because it influences how easily a model can be fine-tuned, how robust it is to pruning, and how it might "forget" tasks during continual learning. This paper moves beyond asking if information is present to asking how sparsely it is stored.

Methods

The authors define $k^*$ as the smallest number of features required to recover 95% of the performance of a "dense" probe (one that uses all available features).

To find $k^*$, they use a fixed search protocol involving:

  1. Feature Ranking: Identifying the most relevant features using statistical t-scores.
  2. L1 Regularization: Using "Lasso" style regression to identify the most efficient subsets of features.
  3. Cross-Model Matching: Using Sparse Autoencoders (SAEs) to see if the specific features identified in one model (e.g., a "Japanese language" feature) correspond to similar features in a model from a different lab.
  4. Controlled Synthetic Experiments: Training small models on purely synthetic tasks to isolate exactly how supervision changes the sparsity of a representation.

Datasets

The study evaluates 19 different LMs (including the Gemma, Llama, Mistral, OLMo, and Qwen families) across 14 core tasks. These tasks include:

  • Dispatch tasks: Language identification, tool selection, and intent classification (e.g., CLINC150, Banking77).
  • Safety/Judgment tasks: Harmfulness detection (WildGuardMix), jailbreak detection, and fact-checking (FEVER).
  • Reasoning/Correctness: Math and code bug detection (ProcessBench, Devign).

Key Insights and Ideas

  • Universal Sparsity Patterns: Despite being trained by different organizations on different data, base models show a striking agreement on which tasks are "narrow" (require few features) and which are "wide" (require many). For example, Language ID always needs ~1 feature, while Harmfulness detection needs hundreds.
  • Supervision as a Compressor: The most "aha" moment comes from synthetic experiments. When a model is supervised on a specific target, it organizes that information into a single feature, even if the logic is complex. Readable information that wasn't explicitly supervised remains "wide," requiring tens of features to extract.
  • Instruction Tuning's Impact: Instruction tuning (IT) selectively reduces $k^*$ for tasks relevant to the tuning (like safety and fact-checking). This suggests that the "chat" version of a model isn't just better at following instructions; it has physically reorganized its internal map to make safety-relevant information more accessible.
Task readout counts and rankings across base models

Key Results

  • Predicting Learning Requirements: The authors found an empirical scaling law: $N^* \propto (k^*)^{0.65}$. This means the sparsity of a task in a "teacher" model can predict how many labeled examples ($N^*$) a "student" model will need to learn that same task.
  • Behavioral Forgetting vs. Linear Readability: In continual learning, a model might "forget" how to perform an old task (the classification head fails). However, the internal $k^*$ measurements show that the information is often still there and just as readable as before; it's the "head," not the "knowledge," that broke.
  • Pruning Vulnerability: "Wide" tasks (high $k^*$) are significantly more vulnerable to weight pruning. If a task's information is spread across many features, losing a random 60% of weights is much more likely to degrade performance than if the information were concentrated in a single feature.
How $k^*$ relates to learning and model maintenance

Limitations and Future Work

  • Upper Bound Nature: The $k^*$ metric is an upper bound based on a specific search protocol; it's possible even smaller subsets exist that the protocol missed.
  • The "Why" of Intermediate Sparsity: While synthetic targets become 1-feature readouts, many real-world tasks in pretrained models have $k^*$ values between 10 and 100. The mechanism that produces these intermediate levels of concentration is still unknown.
  • Intervention Challenges: While $k^*$ is great for diagnosing model changes, the authors' attempts to use it for interventions (like protecting specific weights during pruning) did not yet outperform simple random baselines. This suggests a gap between knowing which features are used and knowing which specific weights sustain them.

Summary updated

Abstract

Dense linear probes assess task readability from language model (LM) representations, but do not directly reveal how many features suffice for prediction. We measure readout sparsity with \(k^*\), the smallest feature count found by a fixed search protocol that recovers 95% of dense-probe AUROC above chance. We evaluate 19 LMs from five labs on 14 core tasks. Ten base models give the nine reliably readable tasks similar feature counts and nearly the same order. This agreement extends to the selected features themselves, which on three tasks show cross-model response correspondence beyond class-mean differences and a label-correlation-matched null. Instruction-tuned deployment selectively reduces \(k^*\), with both weights and input format contributing. In controlled synthetic experiments across 20 models, successfully learned targets admit single-feature readouts, whereas readable controls require tens of features. Reference-model \(k^*\) is associated with label requirements across eight students, following an empirical scaling of \(N^*\propto(k^*)^{0.65}\). After continual learning, saved heads can fail while linear readability remains high, and recovery requires fewer labels than initial task learning under the tested procedures. Pruning reduces linear readability, with wider pre-pruning readouts generally associated with larger losses. These findings distinguish task readability from readout sparsity and establish their complementary value for predicting learning requirements and diagnosing LM changes.
BibTeX
@misc{wan2026how,
  author = {Kaiyang Wan and Yuxia Wang},
  title = {{How Language Models Organize Task Information}},
  year = {2026},
  eprint = {2610.27724},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.27724},
}
Figure