In-context learning (ICL) allows Large Language Models to perform new tasks simply by observing a few demonstrations in their prompt. As these models evolve into Multimodal Large Language Models (MLLMs), the challenge shifts from picking the right words to picking the right combinations of images, text, audio, or even molecular structures.
Recent research suggests that the traditional method of random example selection is insufficient for multimodal tasks. Instead, a new paradigm has emerged: multimodal example selection, which uses cross-modal similarity to retrieve demonstrations that are semantically and structurally aligned with the query.
The Failure of Random Selection
In text-only models, random selection often works reasonably well. However, Xu Yang et al. demonstrated in Exploring Diverse In-Context Configurations for Image Captioning that for vision-language tasks, the choice of image-caption pairs dramatically impacts performance. They found that excessive visual similarity can sometimes lead to model "laziness," while poor alignment causes the model to ignore the visual context entirely.
This "blindness" was further explored by Shuo Chen et al. in Can Multimodal Large Language Models Truly Perform Multimodal In-Context Learning?. The authors discovered that while MLLMs often rely heavily on the textual part of a demonstration for reasoning, the visual component is essential for selecting the most relevant examples from a database. Without visual guidance, the retrieved context often lacks the specific nuances required to solve the task at hand.
The Two-Stage Retrieval Pipeline
To bridge this gap, researchers have converged on a multi-stage retrieval strategy. Rather than relying on a single modality, these systems filter and then refine their selections:
- Visual Filtering: A large pool of candidates is narrowed down by comparing the visual features of the query image to potential demonstrations using embeddings from pre-trained models like CLIP.
- Semantic Reranking: The remaining candidates are reranked based on how well their textual descriptions or metadata match the query's intent.
Yucheng Zhou et al. utilized this two-stage approach in Visual In-Context Learning for Large Vision-Language Models to convert visual demonstrations into "intent-oriented" summaries. Similarly, Shuo Chen et al. proposed the MMICES strategy, showing that this hybrid retrieval significantly outperforms single-modality selection by ensuring the final demonstrations are both visually and linguistically relevant.
Cross-Modal Alignment and Dual Encoding
The intuition behind this approach is that a "good" demonstration acts as a relevant analogy. For the model to apply the logic of an example to a new query, the two must share a common representational space.
- Dual-Encoder Strategies: Dongnan Liu et al. showed in Enhancing Advanced Visual Reasoning Ability of Large Language Models that combining textual and multimodal similarity ensures that retrieved examples share both semantic context and visual characteristics.
- Multi-Stage Matching: In Online In-Context Distillation for Low-Resource Vision Language Models, Zhiqi KANG et al. extended this to include image-to-image, image-to-text, and text-to-text comparisons, overcoming the limitations of generic text-only queries.
This alignment is particularly critical in specialized domains. In medical imaging, Mai A. Shaaban et al. developed MedPromptX, which uses multimodal similarity to retrieve chest X-ray cases that share both visual pathologies and clinical indicators from electronic health records.
Matching Strategy to Task Modality
Not all multimodal tasks require the same retrieval balance. Nan Xu et al. provided a principled analysis in From Introspection to Best Practices, arguing that the optimal selection strategy depends on whether a task is "vision-heavy" or "text-heavy."
For example, tasks involving information extraction from visual layouts (like charts or receipts) benefit most from vision-driven retrieval. Conversely, general visual question answering (VQA) often performs better with text-driven selection. By matching the retrieval modality to the task's primary bottleneck, researchers can significantly boost the reasoning capabilities of frozen models.
Beyond Images: Molecules and Audio
The principles of multimodal selection extend to other complex data types:
- Molecules: Jiatong Li et al. developed ICMA, which uses graph neural networks for structural similarity and lexical search for textual descriptions to help LLMs understand chemical properties.
- Audio: In Retrieval-Augmented Text-to-Audio Generation, Haohe Liu et al. used contrastive multi-modal retrieval to find audio-text pairs that help models generate rare or complex sounds by providing high-quality acoustic context.
By treating example selection as a cross-modal retrieval problem, these frameworks allow general-purpose models to act as domain experts, proving that what you show a model is just as important as how the model is trained.
Key Foundational Work
Image-Grounded Conversations: Multimodal Context for Natural Question and Response Generation (2017) — contributed 1 concept elements to this direction.
Similar Scenes arouse Similar Emotions: Parallel Data Augmentation for Stylized Image Captioning (2021) — contributed 1 concept elements to this direction.