For years, the standard way to adapt a Large Language Model (LLM) to a new task without fine-tuning was "few-shot" prompting—providing three to five examples to nudge the model in the right direction. However, as context windows have expanded from a few thousand to millions of tokens, a new regime has emerged: Many-Shot In-Context Learning (ICL).
By providing hundreds or even thousands of demonstrations, researchers have found that model performance doesn't just plateau; it scales consistently, often rivaling the performance of supervised fine-tuning without a single weight update.
From Pattern Matching to Bayesian Inference
To understand why many-shot learning works, we have to move beyond the idea of the model simply "mimicking" the prompt. Theoretical work by Taiji Suzuki and Tomoya Wakayama in In-Context Learning Is Provably Bayesian Inference suggests that Transformers act as implicit Bayesian reasoners.
In this framework, the model views the prompt as data from which it must infer a "hidden" task. Every additional example reduces the Posterior Variance—the model's uncertainty about what the user actually wants. While a few-shot prompt might leave the model guessing between several similar tasks, a many-shot prompt provides enough statistical density to "lock in" the correct meta-algorithm. This is why Mukai Li and colleagues found in In-Context Learning with Many Demonstration Examples that accuracy continues to climb even as demonstrations scale into the thousands.
Overriding Pre-training Biases
One of the most powerful facets of many-shot ICL is its ability to override a model's existing biases. LLMs are trained on the internet, which means they come with "priors"—pre-existing ideas about how words should relate. If you ask a model to perform a task that contradicts its training (e.g., "label 'happy' as 'negative'"), a few-shot prompt often fails because the pre-training signal is too strong.
However, Rishabh Agarwal and his team demonstrated in Many-Shot In-Context Learning that as the number of demonstrations grows, the in-context signal eventually overpowers the pre-training prior. This allows models to learn highly specialized or even counter-intuitive logic purely from the prompt. This "weight-free" adaptation can reach the same accuracy as parameter-efficient fine-tuning (PEFT), but with much greater flexibility.
The Shift from Selection to Volume
In the few-shot era, "prompt engineering" was an exercise in precision: which five examples are the most representative? In the many-shot regime, the strategy shifts from quality-controlled selection to distributional coverage.
Research in Revisiting In-Context Learning with Long Context Language Models shows that when you have a massive context window, simple random selection of examples becomes as effective as complex retrieval strategies. The sheer volume of data allows the model to capture the underlying distribution of the task. Amanda Bertsch and Uri Alon further noted in In-Context Learning with Long-Context Models that many-shot ICL is significantly more robust to the order of examples, a notorious pain point in few-shot learning.
Scaling Across Modalities and Domains
This scaling behavior isn't limited to text. It appears to be a fundamental property of how Transformer architectures process context:
- Tabular Data: Models like TabPFN, discussed by Felix den Breejen and Sangmin Bae, use ICL to outperform traditional gradient-boosted trees (like XGBoost) by treating rows of a table as in-context demonstrations.
- Vision: Yuanhan Zhang and Kaiyang Zhou have shown that visual models can resolve ambiguities in complex multi-task environments by scaling the number of input-output image pairs in the prompt.
- Robotics: In MetaVLA, Zhantao Yang demonstrates that providing a "context bank" of robotic demonstrations allows models to adapt to new environments and domain shifts on the fly.
The Limits of Contextual Learning
Despite its power, many-shot ICL is not a silver bullet. As prompts grow to include tens of thousands of tokens, models face a "knowing-doing gap." Anian Ruoss and Volodymyr Mnih found in LMAct that even with a million tokens of "know-how," frontier models often struggle to reach expert-level performance in complex imitation learning tasks.
Furthermore, Adrian de Wynter argues in Is In-Context Learning Learning? that while ICL mathematically fits the definition of learning, it remains "brittle," often relying on statistical regularities rather than a deep understanding of task logic. The challenge for the next generation of models is to move from high-volume pattern matching to the robust, structured reasoning required for truly autonomous adaptation.