When large language models (LLMs) expanded their context windows to hundreds of thousands or even millions of tokens, a natural assumption emerged: if a few examples improve performance, providing dozens or hundreds of demonstrations should unlock even greater accuracy. However, empirical evidence across various tasks reveals a counterintuitive phenomenon known as in-context learning (ICL) saturation or non-monotonic few-shot scaling. Instead of a steady upward trajectory, model performance often peaks early and then degrades as more demonstrations are packed into the prompt.
Understanding why this degradation occurs requires looking past the total context length and examining how a Transformer's attention mechanism processes dense, repetitive information.
The Mechanics of Attention Dilution
The primary driver of few-shot saturation is how Transformer architectures allocate attention across long sequences. In a standard prompt, the model must map the relationship between the task instructions, the provided demonstrations, and the final query.
As shown by Zhou et al., scaling the number of examples causes attention dilution. Because attention weights are normalized via the softmax function, expanding the prompt length forces the model to distribute its attention over an increasingly large pool of tokens. Consequently, the model struggles to distinguish the core task patterns from the cumulative noise of excessive demonstrations, a finding echoed by Shu et al.. When the context window is flooded, the signal-to-noise ratio drops, and the model's capacity to isolate and focus on the specific query diminishes.
The In-Context Learning Paradox
A deeper look into model behavior reveals a more fundamental shift in how LLMs process information when overloaded with context. Elazar et al. identified what they term the "ICL Paradox." By testing models with counterfactual inputs, the authors discovered that as the number of demonstrations increases, the model actually becomes less attentive to the specific components of the target input.
Instead of actively reasoning over the new query, the model begins to over-rely on the memorized label distributions and superficial patterns present within the prompt itself. It stops "listening" to the unique problem at hand and defaults to a statistical average of the provided examples. This explains why overall task accuracy can occasionally appear stable even as the model's actual comprehension of the immediate input deteriorates.
Contextual Distraction and In-Context Overfitting
This reliance on surface patterns manifests as contextual distraction or prompt-based overfitting. For instance, in tasks requiring strict adherence to procedural rules, Vahtola et al. observed that adding more examples causes models to prioritize the general stylistic quality and fluency of the output over the precise execution of the requested instruction. The examples distract the model from the underlying logic of the prompt.
Similarly, Xu et al. demonstrated that repeating task demonstrations over long contexts induces a form of model degeneration, akin to overfitting on a dense cluster of redundant information. In complex agent environments, Li noted that dense long-context data triggers "in-context interference," where the sheer volume of expert demonstrations overwhelms the model's reasoning capacity and causes it to deviate from the intended behavior.
Navigating the Saturation Point
Recognizing that more context does not linearly translate to better performance, researchers are shifting focus from volume to efficiency. Rather than blindly scaling prompts, frameworks like those proposed by Gao et al. dynamically search for the optimal stopping point and sequence length for each specific query.
Other approaches, such as the one developed by Drozdov et al., emphasize quality over quantity, demonstrating that a single, carefully selected "difficult" example can outperform a massive block of random demonstrations. Alternatively, Su et al. bypass the context window limitation entirely by processing groups of examples in parallel and merging their predictions via a Naive Bayes framework, avoiding the attention dilution that plagues monolithic prompts.