Abstract
In-Context Learning (ICL) has emerged as a powerful paradigm for Large Language Models (LLMs), enabling them to perform new tasks without parameter updates. However, the performance of ICL is highly sensitive to the choice of demonstration examples. This paper introduces ByCS (Bayesian in-Context example Selection), a novel framework that leverages Bayes’ theorem to identify optimal examples. By employing "inverse inference," ByCS identifies examples that share the highest mutual information with a test input. Our evaluation across text, speech, and visual modalities demonstrates that ByCS consistently outperforms traditional similarity-based selection methods, offering a robust and generalizable solution for multi-modal ICL.
Introduction
Large Language Models (LLMs) exhibit a remarkable ability to learn from a few examples provided in the prompt, a process known as In-Context Learning (ICL). Despite its success, ICL is notoriously "brittle"—the specific selection and ordering of examples can lead to significant variance in model performance. Traditional selection strategies, such as k-Nearest Neighbors (kNN), typically rely on semantic similarity in an embedding space. However, these methods often fail to capture the complex functional relationship between the examples and the specific test query.
As shown in the figure above, ICL is now being applied across diverse modalities, including Automatic Speech Recognition (ASR) and Visual Question Answering (VQA). This paper addresses the challenge of example selection by proposing a unified Bayesian framework that works effectively across these different data types.
The ByCS Framework
The core innovation of this work is the ByCS algorithm, which treats example selection as a Bayesian inference problem. Instead of simple similarity, ByCS seeks to maximize the mutual information between the selected examples and the test input.
Methodology
The selection process involves three primary stages:
- First-round Inference: The model generates an initial "hypothesized" label ($\hat{Y}$) for the test input ($X$) based on a preliminary set of examples.
- Inverse Inference: This is the critical step where the model treats the test input and its hypothesized label as part of the prompt to predict the labels of candidate examples ($C_{label}$) from a datastore. The intuition is that if an example is truly representative of the task for the test input, the test input should conversely be a good predictor for that example.
- Score-based Reranking: Candidate examples are scored based on the text similarity between their actual labels and the labels predicted during inverse inference. The top-ranked examples are then selected for the final ICL prompt.
To maintain computational efficiency, the authors utilize a two-stage approach: a fast pre-selection (e.g., kNN) to narrow the search space, followed by the Bayesian reranking. Additionally, smaller "proxy" models can be used for the inverse inference step without significant loss in selection quality.
Experimental Evaluation
The effectiveness of ByCS was validated across seven datasets spanning three modalities, including text classification (TREC, SST2), text-to-SQL (Spider), ASR error correction (HyPoradise), dialect-specific ASR (RASC863, CORAAL), and knowledge-based VQA (OKVQA).
Key Results
- Superior Performance in Speech: In ASR tasks on the RASC863 dataset, ByCS achieved a 10.25% relative reduction in Word Error Rate (WER) over the KATE+ baseline.
- Consistency in Text and Vision: ByCS consistently outperformed BM25 and LLM-R on open-ended text tasks. In VQA, it showed steady improvements as the number of shots ($k$) increased, demonstrating better scaling than kNN.
- Cross-Model Robustness: The method proved effective across various model architectures, including Whisper, GPT-3.5/4, and Emu2.
Discussion and Insights
The success of ByCS highlights that the "best" examples are those with a strong functional relationship to the query, rather than just semantic proximity. An interesting finding was the "Worse is Better" paradox: using more challenging decoding settings during inverse inference (e.g., removing task-specific prompts) actually improved the selection quality by making it easier for the model to distinguish between high- and low-utility examples.
Limitations
While powerful, ByCS currently assumes independence between candidate examples, ignoring potential inter-example interactions within a prompt. Furthermore, for tasks with extremely short labels (e.g., single-word classification), the similarity-based scoring can become noisy due to sharper score distributions.
Conclusion
ByCS provides a principled, Bayesian approach to example selection that generalizes across text, speech, and vision. By focusing on mutual information through inverse inference, it significantly enhances the robustness and accuracy of in-context learning, paving the way for more reliable multi-modal AI systems.
Summary updated