Provoking Multi-modal Few-Shot LVLM via Exploration-Exploitation In-Context Learning

Cheng Chen, Yunpeng Zhai, Yifan Zhao, Jinyang Gao, Bolin Ding, Jia Li

TL;DR

Few-shot In-Context Learning (ICL) for Large Vision-Language Models typically relies on simple embedding similarity to select examples, ignoring how those examples interact. This paper introduces a novel reinforcement learning framework that treats demonstration selection as a combinatorial optimization problem, using an exploration-exploitation loop to adaptively choose the most complementary and effective set of multi-modal examples.

What Problem Does This Paper Address?

When prompting Large Vision-Language Models (LVLMs), providing a few examples—known as In-Context Learning (ICL)—can dramatically improve performance on novel tasks without fine-tuning. However, how do we choose the best examples?

Current methods usually pick examples that are individually most similar to the user's query (e.g., via cosine similarity of embeddings). The authors identify two major flaws in this approach:

  1. Similarity $\neq$ Utility: Just because an example is close in embedding space doesn't mean it helps the model reason better.
  2. Information Redundancy: Selecting examples independently ignores how they interact. If you pick the top three most similar examples, they might all contain the exact same redundant information, failing to provide diverse context.

Because there are no direct supervision signals showing what the "perfect" combination of examples looks like, the authors reformulate demonstration selection as a combinatorial optimization problem solved via reinforcement learning.

Methods

The proposed framework consists of three main components organized into an iterative Exploration-Exploitation loop:

Figure 2. The pipeline of the proposed framework. It comprises three parts: multi-modal interactive encoding, self-exploration, and self- exploitation. First, the questions and images of queries and candidate demonstrations are encoded to generate multi-modal features, which are subsequently processed through a fusion encoder to output the interactive features. Following this, we employ auto-regressive decoder to generate multiple demonstration combinations that are suitable for the query. These combinations are used in few-shot inference. Finally, the predicted outputs are compared against the ground-truth to get responsive rewards, leveraged to optimize the selection policy.
  1. Multi-modal Interactive Encoding: The query (image + text) and candidate demonstrations are first encoded using ImageBind. A Transformer-based fusion encoder then models the global interactions across all tokens and modalities, capturing cross-modal relationships.
  2. Stochastic Beam Search (Exploration): An auto-regressive decoder selects demonstrations sequentially. Instead of standard greedy decoding (which gets stuck in local optima), the authors implement a stochastic beam search. This allows the system to sample multiple diverse candidate demonstration sets for a given query.
  3. Responsive Reinforcement (Exploitation): The sampled demonstration sets are fed along with the query into the target LVLM. The system evaluates the accuracy of the LVLM's answers to generate a reward signal. Using policy gradient reinforcement learning with query-wise standard normalization, the framework updates the selection policy—increasing the probability of high-performing combinations while suppressing redundant or misleading ones.
Figure 1. The motivation of the proposed method. The method consists of two stages: exploration and exploitation. In the ex- ploration stage, a stochastic beam search is conducted to generate multiple selection strategies for the query. We reason the output with a large language model and comprehensively evaluate these multiple demonstration sets. In the exploitation stage, the policy is optimized based on evaluation, making it more likely to select the optimal combination. These two stages iterate continuously, resulting in seeking the optimal strategy.

Datasets

The framework was evaluated across two prominent LVLMs (Qwen-VL and LLaVA) on four diverse Visual Question Answering (VQA) benchmarks:

  • OKVQA: Requires external knowledge to answer questions about images.
  • TextVQA: Requires reading and reasoning about text present within images.
  • VizWiz: Real-world, often imperfect images raised by visually impaired users.
  • MMStar: A benchmark designed to evaluate core multi-modal capabilities with balanced, challenging samples.

Key Insights and Ideas

  • The Power of Complementarity: The framework often chooses a first demonstration that is highly dissimilar to the query, followed by a second one that is more similar. This balances diversity and similarity, providing a richer context than similarity-only heuristics.
  • Diminishing Returns of Scale: Performance peaks at exactly two demonstrations. Adding more examples increases token overhead and complicates the reinforcement learning optimization space, leading to a performance drop.
Figure 3. Left: VQAScore on OKVQA benchmark of our method with different number of demonstrations. Right: VQAScore on OKVQA benchmark of our method with different temperature val- ues in stochastic beam search.
  • Heuristics Can Mislead: Standard similarity metrics can accidentally force the model to focus on prominent but irrelevant background objects. The learned policy successfully steers the LVLM's attention toward subtle, task-relevant details.

Key Results

The method was evaluated using VQAScore as the primary metric, consistently outperforming zero-shot, random, BM25, and similarity-based retrieval baselines.

  • Qwen-VL Improvements: On OKVQA, the method achieved a VQAScore of 64.7 (outperforming the similarity baseline at 60.6). On VizWiz, it boosted performance from 48.7 to 53.6.
  • LLaVA Improvements: The framework provided an even more massive boost to the weaker LLaVA model, raising its average score across benchmarks from 31.3 (similarity) to 40.8.

Limitations and Future Work

The primary limitation is that this framework currently only explores language-vision multi-modal ICL. As modern large models rapidly integrate more diverse inputs, the authors note that extending this exploration-exploitation framework to video and audio modalities remains an important direction for future work.

Summary updated

Abstract

In-context learning (ICL), a predominant trend in instruction learning, aims at enhancing the performance of large language models by providing clear task guidance and examples, improving their capability in task understanding and execution. This paper investigates ICL on Large Vision-Language Models (LVLMs) and explores the policies of multi-modal demonstration selection. Existing research efforts in ICL face significant challenges: First, they rely on pre-defined demonstrations or heuristic selecting strategies based on human intuition, which are usually inadequate for covering diverse task requirements, leading to sub-optimal solutions; Second, individually selecting each demonstration fails in modeling the interactions between them, resulting in information redundancy. Unlike these prevailing efforts, we propose a new exploration-exploitation reinforcement learning framework, which explores policies to fuse multi-modal information and adaptively select adequate demonstrations as an integrated whole. The framework allows LVLMs to optimize themselves by continually refining their demonstrations through self-exploration, enabling the ability to autonomously identify and generate the most effective selection policies for in-context learning. Experimental results verify the superior performance of our approach on four Visual Question-Answering (VQA) datasets, demonstrating its effectiveness in enhancing the generalization capability of few-shot LVLMs.
BibTeX
@inproceedings{chen2025provoking,
  author = {Cheng Chen and Yunpeng Zhai and Yifan Zhao and Jinyang Gao and Bolin Ding and Jia Li},
  title = {{Provoking Multi-modal Few-Shot LVLM via Exploration-Exploitation In-Context Learning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.00362},
}
Figure