Abstract
We investigate the internal representations of vision-and-language models (VLMs) and how they encode task representations. We consider tasks specified through examples or instructions, using either text or image inputs. Surprisingly, we find that conceptually similar tasks are mapped to similar task vector representations, regardless of how they are specified. Our findings suggest that to output answers, tokens in VLMs undergo three distinct phases: input, task, and answer, a process which is consistent across different modalities and specifications. The task vectors we identify in VLMs are general enough to be derived in one modality (e.g., text) and transferred to another (e.g., image). Additionally, we find that ensembling exemplar and instruction based task vectors produce better task representations. Taken together, these insights shed light on the underlying mechanisms of VLMs, particularly their ability to represent tasks in a shared manner across different modalities and task specifications. Project page: https://task-vectors-are-cross-modal.github.io.
Introduction
Modern Vision-Language Models (VLMs) like LLaVA and Idefics2 exhibit remarkable versatility, yet the internal mechanisms driving this flexibility remain largely opaque. A fundamental question in the field is whether a model's internal representation of a task depends on its input modality—for instance, whether a task specified via text instructions is processed differently than one specified via image examples. This research investigates the existence of "task vectors": compressed mathematical representations that serve as a shared "latent language" for task execution across modalities.
Methodology
The researchers utilize a technique called activation patching (or cross-modal patching) to isolate and transfer these representations. The process involves three primary steps:
- Extraction: Running a "source" prompt (e.g., text-only examples like "France: Paris") through the model and extracting the hidden state vector at a specific intermediate layer.
- Injection: Replacing the internal activation of an "unseen" query (e.g., an image of an Italian flag) with the extracted task vector during its forward pass.
- Evaluation: Measuring the model's ability to produce the correct answer (e.g., "Rome") without any explicit task description in the target prompt.
The study evaluates these mechanisms across six custom tasks (e.g., Country-Capital, Animal-Young, Food-Color) and complex visual question answering (VQA) datasets including VQAv2 and OK-VQA.
Key Results and Insights
Modality-Invariant Task Representations
Empirical analysis using t-SNE visualizations demonstrates that while raw input embeddings cluster by modality, the internal task vectors cluster by the task itself. This suggests that VLMs build a meta-summary of the intended operation that is independent of whether it was triggered by text or images.
Superiority of Patching over Prompting
Activation patching frequently outperforms standard few-shot prompting. On the LLaVA-v1.5 model, patching a text-derived task vector onto an image query achieved 32% accuracy, compared to just 18% for standard prompting. This indicates that patching provides a "cleaner" signal, mitigating the model's tendency to simply caption visual inputs. Furthermore, patching is significantly more efficient, reducing runtime by up to 11x for long contexts.
The Three-Phase Evolution
The study identifies a consistent three-phase progression of tokens across model layers:
- Input Phase: Processing of raw pixels or text tokens.
- Task Phase: Compression of context into a task-specific meta-summary (e.g., internal states decoding to concepts like "currency").
- Answer Phase: Execution of the task to generate the final output.
Conclusion
This work demonstrates that VLMs build a shared, cross-modal latent language for tasks. These task vectors are not only modality-blind but also more efficient and effective for task execution than traditional prompting. These findings suggest that VLMs often re-use functional logic learned during text-only pre-training to solve visual tasks. Future research is required to determine if these shared representations hold up for more complex, multi-step reasoning or creative generation.
Summary updated