The Data Efficiency of Few-Shot In-Context Learning

71 concepts66 papers4 proposals

In traditional machine learning, adapting a model to a new task typically requires thousands of labeled examples. However, the emergence of Large Language Models (LLMs) has shifted this paradigm toward few-shot learning, where a model adapts to a complex task using only a handful of demonstrations. This efficiency is not merely a convenience; it represents a fundamental shift in how models leverage pre-existing knowledge to solve niche, high-stakes problems where data is scarce.

Calibration Over Training

The primary reason few-shot learning works is that LLMs often already possess the underlying reasoning or domain knowledge required for a task. The few-shot examples serve as a "calibration" mechanism rather than a training phase. Kevin P. Yancey et al. demonstrated this in the context of essay scoring: without calibration examples, GPT-4 performed worse than simple baselines, but with just a few, it achieved performance comparable to established automated writing evaluation systems [Rating Short L2 Essays on the CEFR Scale with GPT-4].

This calibration effect is particularly visible in specialized fields. Yan Hu et al. found that providing task descriptions and a small number of annotated examples significantly improved clinical named entity recognition (NER), allowing models to extract critical health information without massive labeled datasets [Improving large language models for clinical named entity recognition via prompt engineering]. Similarly, Zhaomin Xiao et al. showed that as few as one to five examples significantly boost location prediction accuracy [Analyzing Large Language Models’ Capability in Location Prediction].

The Low-Resource Advantage

The most striking evidence for the effectiveness of few-shot learning appears in "low-resource" scenarios—domains where data is expensive or difficult to obtain. In these settings, LLMs often outperform specialized models that require extensive fine-tuning.

Unlocking Latent Capabilities

Few-shot examples do more than just define a task; they can "unlock" capabilities that the model has been trained to suppress or that are otherwise hidden. Ryan Greenblatt et al. explored this through "password-locked" models, finding that as few as one to four high-quality demonstrations could elicit hidden skills even when the model was explicitly trained to underperform [Stress-Testing Capability Elicitation With Password-Locked Models].

This latent capability makes LLMs highly data-efficient proxy functions. Minae Kwon et al. demonstrated that LLMs acting as reward functions for reinforcement learning could match the accuracy of supervised models trained on ten times as much data [Reward Design with Language Models].

Why Examples Beat Instructions

A recurring theme in recent research is that concrete examples are often more effective than high-level instructions or "personas." Daniel Spokoyny et al. found that in-context learning through few-shot examples significantly improved performance on complex climate policy classification, whereas assigning "expert personas" provided only marginal benefits [Aligning Unstructured Paris Agreement Climate Plans with Sustainable Development Goals].

This suggests that the model's ability to pattern-match from specific instances is more robust than its ability to follow abstract role-playing directives. By providing a minimal set of examples formatted to mimic the model's original pre-training data, researchers can induce specific behaviors—like self-rationalization or complex SQL generation—without the tens of thousands of human-annotated examples typically required [Few-Shot Self-Rationalization with Natural Language Prompts; G.J. Xiong et al.].

Key Foundational Work

Few-Shot Self-Rationalization with Natural Language Prompts (2022) — contributed 1 concept elements to this direction.
Analyzing Commonsense Emergence in Few-shot Knowledge Models (2021) — contributed 1 concept elements to this direction.
GPT Understands, Too (2021) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_few_shot_learning_effectiveness_9144a8f5,
  title = {few-shot learning effectiveness},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/few-shot-learning-effectiveness/rdr_9144a8f519804306bcb0364e362301fd},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_few_shot_learning_effectiveness_9144a8f5_v1,
  title = {few-shot learning effectiveness},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/few-shot-learning-effectiveness/rdr_9144a8f519804306bcb0364e362301fd/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake