In traditional machine learning, adapting a model to a new task typically requires thousands of labeled examples. However, the emergence of Large Language Models (LLMs) has shifted this paradigm toward few-shot learning, where a model adapts to a complex task using only a handful of demonstrations. This efficiency is not merely a convenience; it represents a fundamental shift in how models leverage pre-existing knowledge to solve niche, high-stakes problems where data is scarce.
Calibration Over Training
The primary reason few-shot learning works is that LLMs often already possess the underlying reasoning or domain knowledge required for a task. The few-shot examples serve as a "calibration" mechanism rather than a training phase. Kevin P. Yancey et al. demonstrated this in the context of essay scoring: without calibration examples, GPT-4 performed worse than simple baselines, but with just a few, it achieved performance comparable to established automated writing evaluation systems [Rating Short L2 Essays on the CEFR Scale with GPT-4].
This calibration effect is particularly visible in specialized fields. Yan Hu et al. found that providing task descriptions and a small number of annotated examples significantly improved clinical named entity recognition (NER), allowing models to extract critical health information without massive labeled datasets [Improving large language models for clinical named entity recognition via prompt engineering]. Similarly, Zhaomin Xiao et al. showed that as few as one to five examples significantly boost location prediction accuracy [Analyzing Large Language Models’ Capability in Location Prediction].
The Low-Resource Advantage
The most striking evidence for the effectiveness of few-shot learning appears in "low-resource" scenarios—domains where data is expensive or difficult to obtain. In these settings, LLMs often outperform specialized models that require extensive fine-tuning.
- Discourse and Argumentation: Guizhen Chen et al. found that LLMs could match the performance of state-of-the-art models fine-tuned on thousands of examples by using as few as ten demonstrations [Exploring the Potential of Large Language Models in Computational Argumentation].
- Specialized Linguistics: In Chinese Spell Checking, Ming Dong et al. showed that LLMs using in-context learning significantly outperform specialized discriminative models when training data is scarce [Rich Semantic Knowledge Enhanced Large Language Models for Few-shot Chinese Spell Checking].
- Scientific Discovery: Yaodong Yang et al. applied few-shot learning to protein directed evolution, demonstrating that fine-tuning on a limited number of sequences enables effective performance even when extensive biological data is unavailable [Boosting In-Silicon Directed Evolution with Fine-Tuned Protein Language Model and Tree Search].
Unlocking Latent Capabilities
Few-shot examples do more than just define a task; they can "unlock" capabilities that the model has been trained to suppress or that are otherwise hidden. Ryan Greenblatt et al. explored this through "password-locked" models, finding that as few as one to four high-quality demonstrations could elicit hidden skills even when the model was explicitly trained to underperform [Stress-Testing Capability Elicitation With Password-Locked Models].
This latent capability makes LLMs highly data-efficient proxy functions. Minae Kwon et al. demonstrated that LLMs acting as reward functions for reinforcement learning could match the accuracy of supervised models trained on ten times as much data [Reward Design with Language Models].
Why Examples Beat Instructions
A recurring theme in recent research is that concrete examples are often more effective than high-level instructions or "personas." Daniel Spokoyny et al. found that in-context learning through few-shot examples significantly improved performance on complex climate policy classification, whereas assigning "expert personas" provided only marginal benefits [Aligning Unstructured Paris Agreement Climate Plans with Sustainable Development Goals].
This suggests that the model's ability to pattern-match from specific instances is more robust than its ability to follow abstract role-playing directives. By providing a minimal set of examples formatted to mimic the model's original pre-training data, researchers can induce specific behaviors—like self-rationalization or complex SQL generation—without the tens of thousands of human-annotated examples typically required [Few-Shot Self-Rationalization with Natural Language Prompts; G.J. Xiong et al.].
Key Foundational Work
Few-Shot Self-Rationalization with Natural Language Prompts (2022) — contributed 1 concept elements to this direction.
Analyzing Commonsense Emergence in Few-shot Knowledge Models (2021) — contributed 1 concept elements to this direction.
GPT Understands, Too (2021) — contributed 1 concept elements to this direction.