TL;DR
Task Vectors (TVs)—compact directions used to steer Large Language Model (LLM) behavior—are not monolithic "black boxes." This paper demonstrates that a single TV can be linearly decomposed into two functionally distinct and interpretable components: Task Recognition (TR), which identifies the label space, and Task Learning (TL), which maps inputs to specific labels. By tuning the weights of these components using Bayesian Optimization, the authors significantly improve steering performance across multiple models and tasks.
What Problem Does This Paper Address?
Task Vectors have emerged as a powerful, parameter-efficient way to modify LLM behavior by simply adding a pre-computed vector to the model's hidden states. However, we typically treat these vectors as "atomized" directions—we know they work, but we don't know how they are organized internally.
This lack of transparency makes it difficult to troubleshoot or optimize TVs. Drawing inspiration from research into In-Context Learning (ICL), the authors investigate whether the functional split seen in ICL—recognizing the task vs. learning the specific mapping—is reflected in the geometry of the Task Vector itself. If we can pull these functions apart, can we build better, more controllable steering tools?
Methods
The authors propose a three-step framework for decomposing and optimizing TVs:
- Extraction: They extract a standard TV by calculating the average difference in hidden states between a task-conditioned prompt (with instructions or examples) and a query-only prompt.
- Decomposition: They define two geometric axes based on the model’s "unembedding" matrix (the final layer that turns vectors into words).
- The TR axis represents the structure shared across all candidate labels for a task.
- The TL axis represents the contrast between the correct label and the others.
- They use a Metric-Aware Projection (a Hessian-based approach) to map these axes from the output layer back to intermediate layers, accounting for how the model's sensitivity changes across layers.
- Optimization: They treat the TV as a combination of TR, TL, and a "residual" (everything else). Using Bayesian Optimization (BO), they search for the optimal weights ($\alpha$ for TR and $\beta$ for TL) that maximize task accuracy on a small validation set.
Datasets
The approach was evaluated on four standard classification datasets:
- SST-2: Sentiment analysis (2 labels).
- SUBJ: Subjectivity classification (2 labels).
- Financial PhraseBank (FP): Sentiment in financial contexts (3 labels).
- AG News: Topic classification (4 labels).
- JF-ICR: A challenging Japanese financial task used to test the limits of the method.
The researchers tested these across several model families, including Llama 3.2 (1B), Llama 3.1 (8B), and various Qwen 3.5 models.
Key Insights and Ideas
- Functional Geometry is Real: The most striking "aha moment" is that these mathematically derived axes actually correspond to human-interpretable functions.
- Distinct Logit Effects: As shown in Figure 3, intervening on the TR component alone increases the model's preference for all valid labels (e.g., both "Positive" and "Negative") relative to the rest of the vocabulary. Intervening on the TL component specifically shifts the preference toward the correct label.
- More Isn't Always Better: The optimization often finds that reducing the TL component while increasing the TR component leads to better accuracy. This suggests that the original TV might sometimes be "over-discriminative" or lack sufficient task-level grounding.
- Structured Search Wins: While random directions can sometimes improve a TV, searching within the TR/TL subspace is far more reliable and efficient, particularly for instruction-based steering.
Key Results
- Massive Accuracy Boosts: Optimized TVs consistently outperformed original TVs. For example, on Llama-3.2-1B for the SST-2 task (Instruction-based), the original TV achieved only 28.9% accuracy, while the TR/TL-optimized version jumped to 84.4%.
- Layer-Wise Success: Improvements were most pronounced in the middle and late layers of the models. In early layers, the task information hasn't yet aligned well with the label geometry, making optimization less effective.
- Generalization: The gains held up on independent evaluation sets (data not seen during the BO process), proving that the optimized weights aren't just overfitted to the search data.
Limitations and Future Work
- Starting Point Dependency: The method requires a "reliable" initial Task Vector. On very difficult tasks where the base model fails to produce a coherent TV (like Llama 3.1 on the Japanese JF-ICR task), the optimization has no good starting point and fails to provide gains.
- Linearity Assumption: The authors use a linear decomposition, but LLM computation is non-linear. Future work could explore how TR and TL interact non-linearly across layers.
- Mechanistic Gap: While the paper proves the geometry is functional, it doesn't pinpoint exactly which attention heads or neurons are responsible for these roles. Bridging this geometric view with mechanistic interpretability remains an open challenge.
For more details, see the full paper: "Task Vectors Can Be Linearly Decomposed into Interpretable and Controllable Components" (arXiv:2610.41743v1).
Summary updated