Task Vectors are Cross-Modal

Grace Luo, Trevor Darrell, Amir Bar

Abstract

We investigate the internal representations of vision-and-language models (VLMs) and how they encode task representations. We consider tasks specified through examples or instructions, using either text or image inputs. Surprisingly, we find that conceptually similar tasks are mapped to similar task vector representations, regardless of how they are specified. Our findings suggest that to output answers, tokens in VLMs undergo three distinct phases: input, task, and answer, a process which is consistent across different modalities and specifications. The task vectors we identify in VLMs are general enough to be derived in one modality (e.g., text) and transferred to another (e.g., image). Additionally, we find that ensembling exemplar and instruction based task vectors produce better task representations. Taken together, these insights shed light on the underlying mechanisms of VLMs, particularly their ability to represent tasks in a shared manner across different modalities and task specifications. Project page: https://task-vectors-are-cross-modal.github.io.

Introduction

Modern Vision-Language Models (VLMs) like LLaVA and Idefics2 exhibit remarkable versatility, yet the internal mechanisms driving this flexibility remain largely opaque. A fundamental question in the field is whether a model's internal representation of a task depends on its input modality—for instance, whether a task specified via text instructions is processed differently than one specified via image examples. This research investigates the existence of "task vectors": compressed mathematical representations that serve as a shared "latent language" for task execution across modalities.

Conceptual diagram showing how different modalities (text instructions, text examples, image examples) map to a shared latent space of task representations.

Methodology

The researchers utilize a technique called activation patching (or cross-modal patching) to isolate and transfer these representations. The process involves three primary steps:

  1. Extraction: Running a "source" prompt (e.g., text-only examples like "France: Paris") through the model and extracting the hidden state vector at a specific intermediate layer.
  2. Injection: Replacing the internal activation of an "unseen" query (e.g., an image of an Italian flag) with the extracted task vector during its forward pass.
  3. Evaluation: Measuring the model's ability to produce the correct answer (e.g., "Rome") without any explicit task description in the target prompt.

The study evaluates these mechanisms across six custom tasks (e.g., Country-Capital, Animal-Young, Food-Color) and complex visual question answering (VQA) datasets including VQAv2 and OK-VQA.

t-SNE clustering of context embeddings vs. task vectors (left) and the cross-modal patching architecture (right).

Key Results and Insights

Modality-Invariant Task Representations

Empirical analysis using t-SNE visualizations demonstrates that while raw input embeddings cluster by modality, the internal task vectors cluster by the task itself. This suggests that VLMs build a meta-summary of the intended operation that is independent of whether it was triggered by text or images.

Superiority of Patching over Prompting

Activation patching frequently outperforms standard few-shot prompting. On the LLaVA-v1.5 model, patching a text-derived task vector onto an image query achieved 32% accuracy, compared to just 18% for standard prompting. This indicates that patching provides a "cleaner" signal, mitigating the model's tendency to simply caption visual inputs. Furthermore, patching is significantly more efficient, reducing runtime by up to 11x for long contexts.

The Three-Phase Evolution

The study identifies a consistent three-phase progression of tokens across model layers:

  1. Input Phase: Processing of raw pixels or text tokens.
  2. Task Phase: Compression of context into a task-specific meta-summary (e.g., internal states decoding to concepts like "currency").
  3. Answer Phase: Execution of the task to generate the final output.

Conclusion

This work demonstrates that VLMs build a shared, cross-modal latent language for tasks. These task vectors are not only modality-blind but also more efficient and effective for task execution than traditional prompting. These findings suggest that VLMs often re-use functional logic learned during text-only pre-training to solve visual tasks. Future research is required to determine if these shared representations hold up for more complex, multi-step reasoning or creative generation.

Summary updated

Abstract

We investigate the internal representations of vision-and-language models (VLMs) and how they encode task representations. We consider tasks specified through examples or instructions, using either text or image inputs. Surprisingly, we find that conceptually similar tasks are mapped to similar task vector representations, regardless of how they are specified. Our findings suggest that to output answers, tokens in VLMs undergo three distinct phases: input, task, and answer, a process which is consistent across different modalities and specifications. The task vectors we identify in VLMs are general enough to be derived in one modality (e.g., text) and transferred to another (e.g., image). Additionally, we find that ensembling exemplar and instruction based task vectors produce better task representations. Taken together, these insights shed light on the underlying mechanisms of VLMs, particularly their ability to represent tasks in a shared manner across different modalities and task specifications. Project page: https://task-vectors-are-cross-modal.github.io.
BibTeX
@misc{luo2024task,
  author = {Grace Luo and Trevor Darrell and Amir Bar},
  title = {{Task Vectors are Cross-Modal}},
  year = {2024},
  eprint = {2410.22330},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.22330},
}
Figure