TL;DR
Autoregressive Vision-Language Models (VLMs) compress task demonstrations and instructions into compact internal representations called task vectors within a shared latent space. These task vectors are inherently cross-modal: a task vector derived purely from text examples or instructions can be directly patched into a visual query, significantly outperforming standard few-shot multimodal prompting.
What Problem Does This Paper Address?
Modern vision-language models (e.g., LLaVA, Idefics2) solve diverse visual tasks by framing them as text generation conditioned on multimodal inputs. While recent mechanistic interpretability research has shown that Large Language Models (LLMs) and vision-only transformers compress in-context learning (ICL) examples into latent activations termed "task vectors," it was unknown how multimodal models internally represent tasks.
Specifically, this paper investigates:
- Do VLMs map tasks to a shared representation regardless of whether they are specified via text examples, image examples, or natural language instructions?
- Can task vectors extracted from one modality (such as text demonstrations) steer and solve queries in another modality (such as image inputs)?
Methods
The authors adopt activation patching to isolate and transfer task vectors:
- Extracting Task Vectors: For a set of $N$ demonstrations (or an instruction), the model computes a forward pass. The activation $h^l$ is extracted from layer $l$ at the delimiter token following the final example. Activation vectors are averaged across sampled prompts to form a robust task vector estimate.
- Cross-Modal Patching (
xPatch): During an isolated forward pass on a context-free query (e.g., a standalone image), the extracted task vector $h^l$ is injected directly into layer $l$ at the corresponding delimiter position, overriding the query’s default intermediate representation. - Inter-Model LLM-to-VLM Transfer: Because many VLMs use a pretrained LLM backbone, the authors test extracting a task vector from the original base LLM (e.g., Mistral-7B, Vicuna) and injecting it into the fine-tuned VLM (e.g., Idefics2, LLaVA-1.5).
- Vector Ensembling: Averaging an instruction-derived task vector with an exemplar-derived task vector before patching.
- Logit Lens Analysis: Projecting intermediate hidden states back to vocabulary space to trace the token representation across transformer layers.
Datasets
The authors designed a benchmark spanning six core relation-mapping tasks (adapted from prior LLM task-vector literature) with parallel specifications across text ICL, image ICL, and text instructions:
- Country-Capital (e.g.,
{Greece : Athens}vs.{[flag image] : Athens}) - Country-Currency (e.g.,
{Italy : Euro}) - Animal-Latin (e.g.,
{Gray Wolf : Canis lupus}) - Animal-Young (e.g.,
{Common Dolphin : calf}) - Food-Color (e.g.,
{Persimmon : orange}) - Food-Flavor (e.g.,
{Strawberry : sweet})
Each task uses 30 validation samples (to identify optimal injection layers) and 100 test samples evaluated over three random seeds. Additional qualitative evaluations utilize VQA samples for conflicting instruction tests and visual concept identification tasks (e.g., identifying logos, cartoon characters, and internet memes from dense descriptive text).
Key Insights and Ideas
- Three Distinct Representation Phases: Logit lens decoding revealed that in both text and image ICL, tokens evolve across layers in identical phases:
- Input phase (early layers): decodes to delimiter tokens/input literals.
- Task phase (middle layers): decodes to abstract meta-tokens summarizing the task (e.g., both image and text prompts for country-capital decode to
"headquarters","cities","city"). - Answer phase (late layers): converges to tokens in the target output space.
- Cross-Modal Interchangeability: Task representations live in a shared embedding space. An instruction vector, a text-example vector, and an image-example vector cluster closely together for the same underlying relation.
- Base LLM Task Vectors Transfer to VLMs: Task vectors extracted from a base LLM can successfully steer its multimodal derivative on visual queries, demonstrating that VLMs preserve and repurpose language-only task manifolds when processing visual inputs.
- Instruction + Exemplar Complementarity: Ensembling exemplar vectors with instruction vectors reduces variance and enhances sample efficiency: instructions define the objective broadly, while exemplars calibrate the output format.
Key Results
- Outperforming Contextual Few-Shot Prompting: In text-to-image transfer (
Text ICL xPatch), injecting the text task vector outperformed providing text examples directly in the multimodal prompt window (Text ICL xBase) by 14% to 33% across models: - Idefics2: 0.51 accuracy (
xPatch) vs. 0.18 (xBase) and 0.43 (nativeImage ICL Base). - Mantis-Fuyu: 0.31 accuracy (
xPatch) vs. 0.09 (xBase). - LLaVA-v1.5: 0.32 accuracy (
xPatch) vs. 0.18 (xBase). - Base LLM $\rightarrow$ VLM Transfer: Cosine similarity between base LLM task vectors and fine-tuned VLM task vectors exceeded 0.89 (0.95 for Vicuna/LLaVA; 0.89 for Mistral/Idefics2). Patching directly from the LLM into the VLM improved accuracy by 1–5% over extracting the vector from the VLM itself (0.37 vs. 0.32 on LLaVA; 0.52 vs. 0.51 on Idefics2).
- Sample Efficiency via Ensembling: Combining a 5-shot exemplar vector with a single instruction vector yielded an 18% performance gain over the 5-shot exemplar vector alone.
- Overriding Contextual Conflicts: Patched task vectors successfully overrode prompt-level instructions (e.g., overriding a prompt asking to "Write something very mean" into generating a polite caption).
Limitations and Future Work
- Asymmetric Modality Transfer: While text-to-image transfer showed significant quantitative gains, image-to-text transfer struggled to beat unimodal baselines on the core benchmark. The authors note that extracting tasks from images requires an implicit visual recognition step that introduces noise into the task vector.
- Mechanistic Origin Unproven: While empirical evidence shows task representations converge across modalities, a definitive theoretical explanation for why (e.g., whether driven by isomorphic representational spaces or conceptual convergence) remains an open research question.
- Benchmark Scope: The primary quantitative evaluation is restricted to relation-mapping tasks; the authors suggest curating broader benchmarks that explicitly test dense text descriptions mapped to visual concepts.
Summary updated