VL-ICL Bench: The Devil in the Details of Benchmarking Multimodal In-Context Learning

Yongshuo Zong, Ondrej Bohdal, Timothy M. Hospedales

Introduction

Large Language Models (LLMs) have demonstrated a remarkable capacity for in-context learning (ICL)—the ability to perform new tasks based solely on a few examples provided in the prompt. While Vision Large Language Models (VLLMs) are expected to inherit this capability, current evaluation methods often fail to distinguish between true task learning and simple format mimicry. Standard benchmarks like Visual Question Answering (VQA) or image captioning primarily measure a model's zero-shot capabilities or its ability to adapt to a specific answer style. This paper introduces VL-ICL Bench, a rigorous benchmark designed to evaluate whether VLLMs can truly induce rules, bind novel concepts, and reason across interleaved multimodal inputs.

Methodology

To move beyond "format learning," the researchers implemented several novel evaluation protocols:

  1. Semantic Evaluation: Using an LLM-as-a-judge to assess answers based on meaning rather than exact string matching. This revealed that much of the perceived ICL gain in traditional benchmarks is merely formatting.
  2. ICL Efficiency Metric: A new metric that calculates the area under the accuracy curve relative to the zero-shot baseline, quantifying how effectively a model improves with additional examples.
  3. Modality Ablation: Comparing multimodal performance against text-only versions of the same tasks to identify whether failures stem from reasoning deficits or visual perception bottlenecks.
  4. Context Analysis: Testing techniques like "SelfExtend" to determine if context window limitations are the primary cause of performance degradation in many-shot scenarios.

The VL-ICL Benchmark

The VL-ICL Bench comprises 10 distinct tasks categorized into Image-to-Text (I2T) and Text-to-Image (T2I) domains. These tasks require the model to perform complex operations such as binding synthetic names to novel objects, inducing mathematical rules from visual symbols, and generating images based on newly learned attributes.

Overview of the 10 tasks included in the VL-ICL Bench, spanning image-to-text and text-to-image modalities.

Specific tasks include:

  • Operator Induction: Models must deduce mathematical operations (e.g., addition) from visual examples.
  • Fast Counting: Binding novel names (e.g., "Blicket") to specific counts of objects.
  • TextOCR: Identifying and reading text within specific highlighted regions.
  • CoBSAT: Inducing latent variables from examples to guide image generation.
Example of the Fast Counting task where the model must bind novel names like 'Blicket' and 'Dax' to specific quantities of animals.

Key Insights

The benchmark reveals several critical findings about the current state of VLLMs:

  • The Format Trap: Performance gains on standard benchmarks are often illusory, driven by the model learning the expected output format rather than the underlying task logic.
  • Perception as a Bottleneck: Performance significantly improves when images are replaced with text descriptions, suggesting that the vision-language connector often fails to transmit necessary fine-grained details.
  • Non-Monotonic Scaling: Unlike text-only models, VLLMs often show degraded performance as the number of shots increases. The high token cost of images quickly exhausts context windows, leading to task "confusion."
  • Emergence at Scale: Multimodal ICL appears to be an emergent property. For instance, in the LLaVA-OneVision series, only the 72B model showed significant ICL improvements, while smaller versions remained at baseline levels.

Results and Conclusion

LLaVA-OneVision 72B and GPT-4V are the current leaders in multimodal ICL. Notably, LLaVA-OneVision 72B often surpasses GPT-4V in tasks requiring precise perception. However, the study also highlights that a high zero-shot score does not necessarily predict strong ICL performance; some models remain "stubborn" despite provided examples.

The benchmark underscores the need for improved vision-language connectors and architectures capable of handling the long contexts required for many-shot multimodal learning. Future research should focus on maintaining high-fidelity perception during complex reasoning tasks.

Summary updated

Abstract

Large language models (LLMs) famously exhibit emergent in-context learning (ICL) -- the ability to rapidly adapt to new tasks using few-shot examples provided as a prompt, without updating the model's weights. Built on top of LLMs, vision large language models (VLLMs) have advanced significantly in areas such as recognition, reasoning, and grounding. However, investigations into \emph{multimodal ICL} have predominantly focused on few-shot visual question answering (VQA), and image captioning, which we will show neither exploit the strengths of ICL, nor test its limitations. The broader capabilities and limitations of multimodal ICL remain under-explored. In this study, we introduce a comprehensive benchmark VL-ICL Bench for multimodal in-context learning, encompassing a broad spectrum of tasks that involve both images and text as inputs and outputs, and different types of challenges, from {perception to reasoning and long context length}. We evaluate the abilities of state-of-the-art VLLMs against this benchmark suite, revealing their diverse strengths and weaknesses, and showing that even the most advanced models, such as GPT-4, find the tasks challenging. By highlighting a range of new ICL tasks, and the associated strengths and limitations of existing models, we hope that our dataset will inspire future work on enhancing the in-context learning capabilities of VLLMs, as well as inspire new applications that leverage VLLM ICL. The code and dataset are available at https://github.com/ys-zong/VL-ICL.
BibTeX
@inproceedings{zong2025vl,
  author = {Yongshuo Zong and Ondrej Bohdal and Timothy M. Hospedales},
  title = {{VL-ICL Bench: The Devil in the Details of Benchmarking Multimodal In-Context Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=cpGPPLLYYx},
}
Figure