Adding worked examples to a prompt (few-shot prompting) sometimes makes a language model worse than giving it no examples at all. Benchmark papers report this repeatedly but do not explain it. This note proposes an explanation borrowed from the statistics of mixture models. View the model's answer as a blend of two predictive distributions: what it already believes (the zero-shot prior) and what the demonstrations suggest. Learning from demonstrations then amounts to estimating a weighting function that decides, for each input, how much to trust the new evidence over the prior. Mixture-model theory says this weighting is estimated very slowly in one specific regime: when the demonstrated behavior is close to, but not identical with, what the model already does. In that regime the blended prediction can fit the demonstrations themselves quickly while the weighting stays badly estimated. A poorly estimated weighting does little damage on inputs that resemble the demonstrations. It does real damage on query inputs that do not.
The claim fits in two sentences. Few-shot degradation is an in-context instance of the "density–parameter gap": the overall predicted distribution converges fast, while the parameters of the input-dependent weighting converge at a rate penalized by how little the new task differs from the old one. This predicts that degradation concentrates at intermediate task discrepancy and on queries outside the region the demonstrations cover. Neither prediction follows from format confusion or from existing linear-attention accounts, so both can be tested directly.
Core Claim
The statistical engine comes from On Minimax Estimation of Parameters in Softmax-Contaminated Mixture of Experts. That paper studies a frozen pre-trained expert combined with a trainable "prompt expert" through an input-dependent softmax gate, where a gate is a small model that outputs mixing weights. It proves that when the prompt expert differs from the frozen one by a small discrepancy $\Delta$, the expert parameters are estimated at rate $n^{-1/2}\|\Delta\|^{-1}$ and the gating parameters at $n^{-1/2}\|\Delta\|^{-2}$. Here $n$ is the sample size.
That paper analyzes gradient-based adaptation. The proposal reinterprets its setting for in-context learning. The frozen expert becomes the model's zero-shot conditional density $f_0(y\mid x)$. The prompt expert becomes the task the $K$ demonstrations are drawn from. The gate becomes the implicit, input-dependent trust the model places in the context.
Treating the pre-trained predictive as one expert in a mixture loses no generality. Approximations of conditional probability density functions in Lebesgue spaces via mixture of experts models proves that softmax-gated mixtures of location-scale experts are dense in $L_p([0,1]^d \times \mathbb{R}^q)$. Any conditional density on a compact input domain can therefore be written this way to arbitrary precision. In this proposal that result serves as a constraint and justification rather than a mechanism: it moves the question from expressivity to estimation.
The weighting matters most when the query $x^\ast$ lies outside the support of the demonstration inputs. On the demonstration inputs, a wrong gate and a wrong expert can partially cancel. That cancellation is the same non-identifiability that Is Temperature Sample Efficient for Softmax Gaussian Mixture of Experts? and Statistical Advantages of Perturbing Cosine Router in Mixture of Experts trace to linear dependencies among partial derivatives. The fitted density on in-support inputs is therefore accurate. Extrapolating to $x^\ast$, by contrast, depends on the gate parameters themselves, which carry the $\|\Delta\|^{-2}$ penalty.
A rough calculation makes the prediction concrete. The excess off-support loss from gate error is roughly the gate error times the expert separation, $K^{-1/2}\|\Delta\|^{-2}\cdot\|\Delta\|$. The benefit of adapting correctly scales like the KL divergence between nearby experts, roughly $\|\Delta\|^2$. Comparing the two suggests that the number of demonstrations needed before few-shot beats zero-shot off-support grows like a steep inverse power of $\|\Delta\|$:
$$K^\ast(\Delta) \propto \|\Delta\|^{-a}, \quad a>0,$$
where $a$ is an exponent to be measured, not assumed; the rough calculation suggests $a$ near 6. Few-shot gain should therefore be non-monotone in $\|\Delta\|$. When $\Delta \to 0$, the prior is already right, so there is nothing to gain. When $\Delta$ is large, the tasks are distinguishable, and the paper above shows the rate returns to $\tilde{O}(n^{-1/2})$. At intermediate $\Delta$, degradation should be worst.
Research Program
The program has three parts. The first is a controlled synthetic world where every quantity is known. The second is a separation test between statistical and architectural explanations. The third transfers the predictions to real models.
Synthetic world. Each pre-training task is a softmax-gated Gaussian mixture-of-experts conditional density on $[0,1]^d$. Each test task contaminates the prior predictive with one extra expert offset by a controlled $\Delta$, using an input-dependent gate whose active region can be placed inside or outside the demonstration input distribution.
Separation test. Small transformers with a mixture-density output head are trained to do in-context conditional density estimation over this task family. They are then compared against the explicit maximum-likelihood estimator of the contaminated mixture, given the same $K$ samples. If the explicit estimator shows the same off-support, intermediate-$\Delta$ degradation, the phenomenon is statistical and no architecture escapes it. If only the transformer degrades, the cause is architectural, and the linear-attention account of Fine-Tuning Without Forgetting In-Context Learning: A Theoretical Analysis of Linear Attention Models becomes the leading explanation.
Transfer and intervention. On real models, an empirical proxy for $\|\Delta\|$ is the zero-shot model's average surprise (negative log-likelihood) on the demonstration answers. Off-support is operationalized as demonstrations drawn from a different subject or domain than the query. The intervention is an inference-time combiner. It mixes the zero-shot and few-shot predictive distributions with a weight estimated by leave-one-out likelihood on the demonstrations. The weight is shrunk toward the prior by an amount that grows as the proxy $\|\Delta\|$ shrinks, which is what the gate-rate theory implies.
The gate in this combiner uses an activation before temperature scaling. Is Temperature Sample Efficient for Softmax Gaussian Mixture of Experts? proves this choice restores polynomial rates where plain temperature-softmax gates fall to $O(1/\log n)$. Here it supplies a method, not just motivation.
Baselines
Zero-shot and few-shot references. The naive reference is plain zero-shot prediction, and the second is plain $K$-shot prediction. Both use the same model and prompt template, so their difference is the quantity under study. On real benchmarks the comparison to beat is whichever of the two is better per subject, chosen in hindsight on held-out subjects. That oracle selector is a ceiling for any method that only decides whether to use demonstrations at all.
Bayes-optimal posterior. In the synthetic world, the strongest competitor is the Bayes-optimal posterior predictive under the true task prior, computed exactly by enumerating or sampling the mixture. Treating in-context learning as implicit Bayesian inference is the most economical rival explanation. Averaged over tasks drawn from that prior, a correct posterior cannot be worse than the prior in expected log loss. The question is therefore whether degradation for specific intermediate-$\Delta$ tasks persists even for the oracle. If it does, the mixture view adds nothing beyond Bayes. If it does not, the transformer is failing to implement the posterior in exactly the regime the gate theory flags.
Contaminated-mixture estimator. The explicit maximum-likelihood estimator of the contaminated mixture is the second strong synthetic baseline. It isolates the statistical part of the claim.
Fine-tuning methods. On real models, the strongest existing intervention is value-only fine-tuning from Fine-Tuning Without Forgetting In-Context Learning. That paper reports a +6.59% zero-shot gain on MMLU Humanities for Qwen2.5-3B-Instruct, with "highly stable" 7-shot out-of-distribution STEM accuracy (no exact number reported), against a −4.07% 7-shot STEM drop for full fine-tuning. The proposal will reproduce both arms on the same model, data split, and LoRA rank before adding the combiner on top.
For translation, the competitor is demonstration-aware training from Towards Demonstration-Aware Large Language Models for Machine Translation. It reports a medical-domain COMET of 85.10 versus 83.30 for vanilla zero-shot ALMA-7B. The combiner will be evaluated as a training-free alternative on the same domains.
Fairness and the beating criterion. All methods get the same demonstrations, the same decoding, and the same tuning budget for the shrinkage schedule, a single hyperparameter selected on held-out subjects. Beating the strongest baseline means two things. On MMLU, off-support accuracy must exceed the hindsight-selected better-of-zero-and-few-shot without losing in-support accuracy. On top of value-only fine-tuning, the combiner must recover more of the off-support 7-shot gap than the fine-tuning alone does.
Ablation. The key ablation removes the $\Delta$-dependent shrinkage and uses a constant mixing weight. If constant mixing performs as well, the gate-rate theory contributes nothing practical.
Why not just…?
Why not just attribute it to formatting or chat-template confusion? C-Eval and MILU do attribute few-shot harm in instruction-tuned models to conversational overfitting, and some real-benchmark degradation surely is formatting. However, the synthetic world here has no template at all. The format explanation also predicts no dependence on $\|\Delta\|$ or on whether the query is in-support. The real-model analysis will report degradation as a function of the surprise proxy, at a fixed template, so the format effect appears as a constant offset rather than a curve.
Why not just use the concept-shift theory? Generalization vs Specialization under Concept Shift derives non-monotone in-context risk for ridge regression when demonstrations and queries follow differently aligned rules. Its mechanism requires the demonstrations' rule to differ from the query's rule. In the synthetic tasks here, demonstrations and query share one task, so that mechanism is switched off. If degradation still appears off-support at intermediate $\Delta$, concept shift cannot be the cause. If degradation vanishes once the rules match, the concept-shift theory wins and this proposal's mechanism is refuted.
Why not just call it implicit Bayesian inference? This is the most serious threat, because a correct posterior is itself a mixture with input-dependent weights. The exact Bayes baseline above settles it. If the oracle posterior degrades in the same cells as the transformer, the contribution reduces to an interpretation: it says which tasks are hard, using rate theory, without identifying a new failure. The proposal would then be reframed as a sample-complexity characterization of Bayesian in-context learning.
Why not just fine-tune only the value matrix? Value-only tuning protects the retrieval circuitry when fine-tuning on a new task. But Towards Demonstration-Aware Large Language Models for Machine Translation shows that even LoRA-tuned models can ignore demonstrations or have them backfire. The present account also predicts degradation in untouched pre-trained models whenever $\Delta$ is intermediate. The real-model experiments include base models that were never fine-tuned, which answers whether value-only tuning addresses the same cause.
Proposed Experiments
Synthetic sweep. The first experiment sweeps $\|\Delta\|$ over two orders of magnitude, $K$ from 1 to 256, and the placement of the gate's active region (inside or outside the demonstration support). It records the few-shot minus zero-shot negative log-likelihood at the query for three predictors: the transformer, the explicit contaminated-mixture estimator, and the Bayes oracle.
The claim survives only if all of the following hold. The in-support gap closes quickly at every $\Delta$. The off-support gap is largest at intermediate $\Delta$. The crossover $K^\ast$ follows a power law in $\|\Delta\|$, with the exponent estimated from a log-log fit. The explicit estimator's gating-parameter error tracks the $n^{-1/2}\|\Delta\|^{-2}$ rate from On Minimax Estimation of Parameters in Softmax-Contaminated Mixture of Experts.
A matched-rule control turns concept shift off, as described above. A second control replaces the gate with the activation-before-temperature form. That gate should shorten $K^\ast$ for the explicit estimator, since Is Temperature Sample Efficient for Softmax Gaussian Mixture of Experts? shows plain temperature-softmax gates can stall at $O(1/\log n)$. That paper reports only plots (error "flatlining" versus dropping "linearly" on log-log axes), and no reported numerical result was found on Lacuna for the synthetic baseline.
MMLU transfer test. On MMLU, Qwen2.5-3B-Instruct and its base counterpart are evaluated with $K\in\{1,3,5,7\}$, using demonstrations from the same subject (in-support) or a different subject (off-support). The few-shot gain is binned by the zero-shot surprise proxy. The decisive check is whether off-support gain is lowest in the middle bins, with no such dip for same-subject demonstrations.
C-Eval transfer test. The same protocol runs on C-Eval and C-Eval HARD. That paper reports that few-shot "actually hurt" MOSS and ChatGLM-6B but gives no exact degradation number, so the proposal will measure that number itself. For scale, its reported 5-shot GPT-4 accuracy is 68.7%.
Translation transfer test. On the medical, law, and IT translation domains from the demonstration-aware machine translation paper, the combiner is compared against vanilla and demonstration-aware ALMA-7B in COMET.
What would falsify the proposal. Any of three outcomes would count against it. Off-support degradation might be flat in the surprise proxy. The Bayes oracle might degrade identically to the transformer. The constant-weight ablation might match the $\Delta$-shrunk combiner. The first two would refute the mechanism, and the third would make it practically irrelevant.
References
- On Minimax Estimation of Parameters in Softmax-Contaminated Mixture of Experts
- Approximations of conditional probability density functions in Lebesgue spaces via mixture of experts models
- Is Temperature Sample Efficient for Softmax Gaussian Mixture of Experts?
- Statistical Advantages of Perturbing Cosine Router in Mixture of Experts
- Fine-Tuning Without Forgetting In-Context Learning: A Theoretical Analysis of Linear Attention Models
- Towards Demonstration-Aware Large Language Models for Machine Translation
- C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models
- MILU: A Multi-task Indic Language Understanding Benchmark
- Generalization vs Specialization under Concept Shift
- Measuring Massive Multitask Language Understanding