TL;DR
Few-shot In-Context Learning (ICL) for Large Vision-Language Models typically relies on simple embedding similarity to select examples, ignoring how those examples interact. This paper introduces a novel reinforcement learning framework that treats demonstration selection as a combinatorial optimization problem, using an exploration-exploitation loop to adaptively choose the most complementary and effective set of multi-modal examples.
What Problem Does This Paper Address?
When prompting Large Vision-Language Models (LVLMs), providing a few examples—known as In-Context Learning (ICL)—can dramatically improve performance on novel tasks without fine-tuning. However, how do we choose the best examples?
Current methods usually pick examples that are individually most similar to the user's query (e.g., via cosine similarity of embeddings). The authors identify two major flaws in this approach:
- Similarity $\neq$ Utility: Just because an example is close in embedding space doesn't mean it helps the model reason better.
- Information Redundancy: Selecting examples independently ignores how they interact. If you pick the top three most similar examples, they might all contain the exact same redundant information, failing to provide diverse context.
Because there are no direct supervision signals showing what the "perfect" combination of examples looks like, the authors reformulate demonstration selection as a combinatorial optimization problem solved via reinforcement learning.
Methods
The proposed framework consists of three main components organized into an iterative Exploration-Exploitation loop:
- Multi-modal Interactive Encoding: The query (image + text) and candidate demonstrations are first encoded using ImageBind. A Transformer-based fusion encoder then models the global interactions across all tokens and modalities, capturing cross-modal relationships.
- Stochastic Beam Search (Exploration): An auto-regressive decoder selects demonstrations sequentially. Instead of standard greedy decoding (which gets stuck in local optima), the authors implement a stochastic beam search. This allows the system to sample multiple diverse candidate demonstration sets for a given query.
- Responsive Reinforcement (Exploitation): The sampled demonstration sets are fed along with the query into the target LVLM. The system evaluates the accuracy of the LVLM's answers to generate a reward signal. Using policy gradient reinforcement learning with query-wise standard normalization, the framework updates the selection policy—increasing the probability of high-performing combinations while suppressing redundant or misleading ones.
Datasets
The framework was evaluated across two prominent LVLMs (Qwen-VL and LLaVA) on four diverse Visual Question Answering (VQA) benchmarks:
- OKVQA: Requires external knowledge to answer questions about images.
- TextVQA: Requires reading and reasoning about text present within images.
- VizWiz: Real-world, often imperfect images raised by visually impaired users.
- MMStar: A benchmark designed to evaluate core multi-modal capabilities with balanced, challenging samples.
Key Insights and Ideas
- The Power of Complementarity: The framework often chooses a first demonstration that is highly dissimilar to the query, followed by a second one that is more similar. This balances diversity and similarity, providing a richer context than similarity-only heuristics.
- Diminishing Returns of Scale: Performance peaks at exactly two demonstrations. Adding more examples increases token overhead and complicates the reinforcement learning optimization space, leading to a performance drop.
- Heuristics Can Mislead: Standard similarity metrics can accidentally force the model to focus on prominent but irrelevant background objects. The learned policy successfully steers the LVLM's attention toward subtle, task-relevant details.
Key Results
The method was evaluated using VQAScore as the primary metric, consistently outperforming zero-shot, random, BM25, and similarity-based retrieval baselines.
- Qwen-VL Improvements: On OKVQA, the method achieved a VQAScore of 64.7 (outperforming the similarity baseline at 60.6). On VizWiz, it boosted performance from 48.7 to 53.6.
- LLaVA Improvements: The framework provided an even more massive boost to the weaker LLaVA model, raising its average score across benchmarks from 31.3 (similarity) to 40.8.
Limitations and Future Work
The primary limitation is that this framework currently only explores language-vision multi-modal ICL. As modern large models rapidly integrate more diverse inputs, the authors note that extending this exploration-exploitation framework to video and audio modalities remains an important direction for future work.
Summary updated