Task Vectors Can Be Linearly Decomposed into Interpretable and Controllable Components

Yukiya Horiba, Hakaze Cho, Takahiro YakoharXiv 2026 · arXiv

TL;DR

Task Vectors (TVs)—compact directions used to steer Large Language Model (LLM) behavior—are not monolithic "black boxes." This paper demonstrates that a single TV can be linearly decomposed into two functionally distinct and interpretable components: Task Recognition (TR), which identifies the label space, and Task Learning (TL), which maps inputs to specific labels. By tuning the weights of these components using Bayesian Optimization, the authors significantly improve steering performance across multiple models and tasks.

What Problem Does This Paper Address?

Task Vectors have emerged as a powerful, parameter-efficient way to modify LLM behavior by simply adding a pre-computed vector to the model's hidden states. However, we typically treat these vectors as "atomized" directions—we know they work, but we don't know how they are organized internally.

This lack of transparency makes it difficult to troubleshoot or optimize TVs. Drawing inspiration from research into In-Context Learning (ICL), the authors investigate whether the functional split seen in ICL—recognizing the task vs. learning the specific mapping—is reflected in the geometry of the Task Vector itself. If we can pull these functions apart, can we build better, more controllable steering tools?

Methods

The authors propose a three-step framework for decomposing and optimizing TVs:

  1. Extraction: They extract a standard TV by calculating the average difference in hidden states between a task-conditioned prompt (with instructions or examples) and a query-only prompt.
  2. Decomposition: They define two geometric axes based on the model’s "unembedding" matrix (the final layer that turns vectors into words).
  3. The TR axis represents the structure shared across all candidate labels for a task.
  4. The TL axis represents the contrast between the correct label and the others.
  5. They use a Metric-Aware Projection (a Hessian-based approach) to map these axes from the output layer back to intermediate layers, accounting for how the model's sensitivity changes across layers.
  6. Optimization: They treat the TV as a combination of TR, TL, and a "residual" (everything else). Using Bayesian Optimization (BO), they search for the optimal weights ($\alpha$ for TR and $\beta$ for TL) that maximize task accuracy on a small validation set.
Overview of the TR/TL decomposition and optimization approach.

Datasets

The approach was evaluated on four standard classification datasets:

  • SST-2: Sentiment analysis (2 labels).
  • SUBJ: Subjectivity classification (2 labels).
  • Financial PhraseBank (FP): Sentiment in financial contexts (3 labels).
  • AG News: Topic classification (4 labels).
  • JF-ICR: A challenging Japanese financial task used to test the limits of the method.

The researchers tested these across several model families, including Llama 3.2 (1B), Llama 3.1 (8B), and various Qwen 3.5 models.

Key Insights and Ideas

  • Functional Geometry is Real: The most striking "aha moment" is that these mathematically derived axes actually correspond to human-interpretable functions.
  • Distinct Logit Effects: As shown in Figure 3, intervening on the TR component alone increases the model's preference for all valid labels (e.g., both "Positive" and "Negative") relative to the rest of the vocabulary. Intervening on the TL component specifically shifts the preference toward the correct label.
  • More Isn't Always Better: The optimization often finds that reducing the TL component while increasing the TR component leads to better accuracy. This suggests that the original TV might sometimes be "over-discriminative" or lack sufficient task-level grounding.
  • Structured Search Wins: While random directions can sometimes improve a TV, searching within the TR/TL subspace is far more reliable and efficient, particularly for instruction-based steering.
Isolated interventions show that TR affects label-set preference while TL affects label discrimination.

Key Results

  • Massive Accuracy Boosts: Optimized TVs consistently outperformed original TVs. For example, on Llama-3.2-1B for the SST-2 task (Instruction-based), the original TV achieved only 28.9% accuracy, while the TR/TL-optimized version jumped to 84.4%.
  • Layer-Wise Success: Improvements were most pronounced in the middle and late layers of the models. In early layers, the task information hasn't yet aligned well with the label geometry, making optimization less effective.
  • Generalization: The gains held up on independent evaluation sets (data not seen during the BO process), proving that the optimized weights aren't just overfitted to the search data.

Limitations and Future Work

  • Starting Point Dependency: The method requires a "reliable" initial Task Vector. On very difficult tasks where the base model fails to produce a coherent TV (like Llama 3.1 on the Japanese JF-ICR task), the optimization has no good starting point and fails to provide gains.
  • Linearity Assumption: The authors use a linear decomposition, but LLM computation is non-linear. Future work could explore how TR and TL interact non-linearly across layers.
  • Mechanistic Gap: While the paper proves the geometry is functional, it doesn't pinpoint exactly which attention heads or neurons are responsible for these roles. Bridging this geometric view with mechanistic interpretability remains an open challenge.

For more details, see the full paper: "Task Vectors Can Be Linearly Decomposed into Interpretable and Controllable Components" (arXiv:2610.41743v1).

Summary updated

Abstract

Task Vectors (TVs) provide a compact way to steer large language models, but they are typically treated as atomized directions merely extracted from hidden states, leaving their internal functional structure poorly understood. Motivated by prior work that decomposes in-context learning into Task Recognition (TR) and Task Learning (TL), we ask whether a single TV can likewise be decomposed into functionally meaningful components. We formulate a TR/TL-based decomposition from candidate-label unembeddings, separating each TV into TR, TL, and residual components, and try to optimize the TR and TL coefficients using Bayesian optimization for more effective intervention. We evaluate this formulation across multiple model families, classification tasks, and both instruction- and demonstration-based TVs. Across these settings, TR/TL-based optimization identifies intervention directions that outperform the original TVs. Although optimization over random directions can also improve performance, TR/TL-based search yields more reliable gains, particularly for instruction-based TVs. Moreover, isolated interventions on the decomposed components produce distinct logit-level effects that are consistent with their intended functional roles. Together, these results suggest that TVs are not monolithic steering directions, but can be decomposed into functionally meaningful components that support both interpretation and control.
BibTeX
@misc{horiba2026task,
  author = {Yukiya Horiba and Hakaze Cho and Takahiro Yakoh},
  title = {{Task Vectors Can Be Linearly Decomposed into Interpretable and Controllable Components}},
  year = {2026},
  eprint = {2610.41743},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.41743},
}
Figure