TL;DR
Vision-Language Models (VLMs) often fail at visual in-context learning because they rely on "textual shortcuts" rather than actually learning from visual examples. This paper introduces MIME, a training set that hides task rules to force visual induction, and DTAD, a dropout technique that masks demonstration labels, significantly boosting VLM performance and making an 8B model competitive with frontier models like Claude Opus 5.
What Problem Does This Paper Address?
Large Language Models (LLMs) are famous for "in-context learning" (ICL)—the ability to figure out a new task just by looking at a few examples in a prompt. You might expect Vision-Language Models to do the same with images (vICL), but they consistently struggle.
The authors discovered a "textual shortcut" problem: VLMs are "lazy." If a prompt includes a specific instruction (e.g., "Count the red circles"), the model performs well. However, if given a generic instruction ("Look at these examples and answer the query"), performance plummets. The models aren't failing because they can't see or can't learn; they simply haven't been trained to look at visual demonstrations to find a hidden rule. Most pretraining data lacks the structured (image, label) demonstration format found in text-only data.
Methods
The researchers proposed two primary solutions to break the model's reliance on text:
- MIME (Multi-source Induction from Mute Episodes): This is a task-agnostic training set designed to require "true visual induction." In MIME, the rule linking images to labels is never stated in text. To solve a query, the model must infer the pattern from the provided visual demonstrations.
- DTAD (Demonstration Text Attention Dropout): To further discourage the model from ignoring images, the authors introduced a novel dropout technique. During training, the text labels for some demonstrations are randomly masked (with a probability of $p=0.5$).
By withholding the label, the only way the model can minimize its loss on the final query is by "reading" the demonstration images to understand the underlying task.
Datasets
The authors curated MIME from five diverse sources to ensure the model learns a general ability rather than a specific task:
- Polygon: Rules based on the number of sides.
- Bongard-style: Abstract concepts like shape properties, rotation, and size.
- ARC-style: Transforming input grids to output grids.
- Triangle arithmetic: Mathematical operations using values at vertices.
- CLEVR: Counting and arithmetic with objects in a scene.
For evaluation, they used several established benchmarks, including TrueMICL, SEED-Bench-2, and VL-ICL-Bench, focusing on tasks where the rule is not explicitly stated.
Key Insights and Ideas
- Hiding the Rule is Essential: The authors found that if the task rule is stated during training, the model's visual attention actually drops. To teach a model to "watch and learn," you must keep it in the dark about the rule.
- Stable Visual Pathways: Analysis of the model's internal attention showed that DTAD builds a "stable visual pathway" in the mid-to-late layers of the transformer. This means the model learns a consistent strategy for extracting information from images across different tasks.
- Cross-Modal Transfer: Surprisingly, training on visual demonstrations improved the models' performance on text-only ICL tasks (like GSM8K). This suggests that teaching a model to find patterns in images helps it become a better general-purpose "in-context learner."
- Efficiency Gains: Fine-tuning a relatively small 8B model on MIME allowed it to match or outperform "frontier" models and open-source models that are 30x larger.
Key Results
- Accuracy Boost: Fine-tuning on MIME improved vICL accuracy by 8.1 to 18.0 points across three different model backbones (Qwen3-VL, LLaVA-OneVision, and Granite-Vision).
- Frontier Performance: A Qwen3-VL-8B model trained with DTAD matched the performance of Claude Opus 5 on the TrueMICL benchmark.
- No Regression: Unlike many fine-tuning methods that degrade a model's original abilities, MIME training preserved (and sometimes improved) single-image performance and multi-image reasoning.
- Retraining Success: Adding MIME to the standard instruction-tuning mix of LLaVA-1.5-13B more than doubled its vICL accuracy (from 18.9% to 39.7%).
Limitations and Future Work
- Scale: The study primarily focused on models in the 4B to 13B range using LoRA adapters. The impact of full-parameter training on massive "frontier-scale" models remains to be seen.
- Data Access: Because the training pipelines for top-tier proprietary models are not public, the authors could not test the effect of adding MIME at the very earliest stages of pretraining.
- Synergy: The authors suggest that MIME/DTAD are complementary to inference-time techniques like "task vectors" or "demonstration retrieval," and combining these approaches could lead to even greater gains in the future.
Summary updated