Visual In-Context Learning for Multimodal Foundation Models

This direction is no longer present in the current build. Its last version stays readable and citable here.

Version 1 of 133 papers37 conceptsfrozen 2026-10-04

In the traditional machine learning paradigm, adapting a model to a new task—like identifying a specific type of cancer in a pathology slide or detecting a rare agricultural pest—required "fine-tuning": updating the model's internal weights using thousands of labeled examples.

However, a new generation of Multimodal Large Language Models (MLLMs) is shifting this requirement. Through a mechanism known as Multimodal In-Context Learning (M-ICL), models like GPT-4V, Gemini, and Med-Flamingo can adapt to specialized domains "on the fly." By simply prepending a few labeled examples (image-text pairs) to a query within the prompt, these models can perform complex tasks without a single weight update.

The Mechanism of Interleaved Prompting

The core intuition behind M-ICL is that the model treats the input prompt as a temporary, local dataset. Instead of a single image and a question, the prompt becomes an interleaved sequence of demonstrations.

For instance, Keshav Bimbraw and colleagues demonstrated in GPT Sonograpy: Hand Gesture Decoding from Forearm Ultrasound Images via VLM that providing just a few ultrasound images labeled with hand gestures allows GPT-4o to decode new images with over 70% accuracy. The model isn't "learning" ultrasound physics; it is using its pre-trained visual reasoning to find the mapping between the provided visual patterns and the text labels within the context window.

This process is often described as exemplar-based calibration. As Siyuan Dai and others suggest in Why Text Prevails: Vision May Undermine Multimodal Medical Decision Making, these examples serve as "anchors" that help the model align subtle visual features with specific categorical labels, effectively narrowing the search space of possible interpretations.

Why It Works: Residuals and Visual Anchors

To understand why a generalist model can suddenly become a specialist, it helps to look at how it handles "normality." Guansong Pang and Jiawen Zhu introduced a framework called InCTRL in Toward Generalist Anomaly Detection via In-Context Residual Learning with Few-Shot Sample Prompts. They frame M-ICL as a "residual" calculation: the model looks at the provided "normal" examples and then identifies anomalies by measuring how much a new query deviates from that in-context distribution.

This visual anchoring is often more powerful than linguistic instruction. In the realm of graphic design, Danqing Huang found in DesignProbe that showing the model visual examples of a design style is significantly more effective than providing abstract textual rules. The model's "eyes" are better at capturing harmony, layout, and metaphor from direct pixels than its "ears" are at parsing descriptions of those same concepts.

Bridging the Data Gap in Specialized Domains

M-ICL is particularly transformative for "low-resource" fields where massive datasets are unavailable:

  1. Medicine: Dyke Ferber and Georg Wölflein showed in In-context learning enables multimodal large language models to classify cancer pathology images that MLLMs can match specialized neural networks by intelligently selecting in-context examples. Similarly, Med-Flamingo uses this to perform rare disease diagnosis where only a handful of cases might exist globally.
  2. Agriculture: In Leveraging Vision Language Models for Specialized Agricultural Tasks, Muhammad Arbab Arshad demonstrated that just eight examples can boost a model's F1 score by 30%, bypassing the need for expert-annotated datasets.
  3. Robotics: The FLARE framework by Byeonghwi Kim uses retrieved visual examples to help embodied agents plan tasks in unfamiliar households using 99.5% less training data than traditional methods.

The Limits of In-Context Adaptation

Despite its power, M-ICL is not a silver bullet. Researchers have identified several "performance boundaries":

  • The Vision-Text Gap: Siyuan Dai noted that in some medical tasks, adding images actually confuses the model, which performs better on text alone. This suggests that the visual modality can sometimes introduce noise that the in-context examples cannot fully filter.
  • Prompt Brittleness: Xiaoqi Zhao and colleagues observed in Inspiring the Next Generation of Segment Anything Models that models like SAM 2 show high sensitivity to the specific examples chosen, making the "selection" of examples as important as the model itself.
  • Cultural Blind Spots: The CROPE benchmark by Malvina Nikandrou reveals that for culture-specific concepts, even providing "cheat sheets" of visual examples often fails to bridge the gap if the model's underlying pre-training is too Western-centric.

Scaling via Retrieval

To move beyond the limited context window of a single prompt, researchers are turning to Retrieval-Augmented Multimodal Models. Weijia Shi developed RA-CM3, which automatically retrieves relevant multimodal documents from an external database to use as in-context examples. This allows the model to "look up" the necessary context for any query, effectively giving it an infinite, dynamic memory that it can use to steer its behavior in real-time.

Papers in this version

History & citationVersion 1 · 2026-10-04

No longer present in the current build · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_visual_in_context_learning_for_multimoda_072eedf3,
  title = {Visual In-Context Learning for Multimodal Foundation Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/visual-in-context-learning-for-multimodal-foundation-models/rdr_072eedf32e2d442ebceaf62282726e00},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_visual_in_context_learning_for_multimoda_072eedf3_v1,
  title = {Visual In-Context Learning for Multimodal Foundation Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/visual-in-context-learning-for-multimodal-foundation-models/rdr_072eedf32e2d442ebceaf62282726e00/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}