Uncertainty Calibration for Few-Shot Learning

This direction is split into other directions: uncertainty calibration. Its last version stays readable and citable here.

Version 1 of 190 papers98 conceptsfrozen 2026-10-04

In high-stakes domains like medical diagnostics or autonomous driving, a model's ability to say "I don't know" is as important as its ability to predict. However, standard few-shot learning (FSL) models—designed to learn from just one or five examples—are notoriously overconfident. When a model sees a single image of a rare disease, it often produces a 99% confidence score, even if that prediction is wrong.

This research direction focuses on uncertainty calibration: ensuring that a model’s predicted probability (confidence) matches its actual likelihood of being correct. By integrating Bayesian principles and statistical bias reduction, researchers are transforming few-shot learners from "confidently wrong" black boxes into reliable decision-making tools.

The Problem of Overconfidence in Low-Data Regimes

Traditional deep learning models rely on point estimates—single values for network weights that maximize likelihood. As Li Fei-Fei and colleagues noted in their seminal work on one-shot learning, these techniques frequently fail to generalize when data is scarce. In the few-shot setting, the model has so little evidence that it "overfits" to the noise or specific quirks of the support set, leading to extreme confidence scores.

This is not just a classification problem; it affects Large Language Models (LLMs) too. Hanlin Zhang observed a performance-reliability trade-off where adding more examples to a prompt improves accuracy but actually worsens calibration, making the model more "confidently wrong" before it sees enough context to stabilize.

Bayesian Meta-Learning: Distributions Over Weights

The most direct way to fix overconfidence is to stop treating neural network weights as fixed numbers. Instead, Bayesian meta-learning treats them as probability distributions.

Gaussian Processes as the "Inner Loop"

A particularly elegant solution is to replace the final layer of a neural network with a Gaussian Process (GP). GPs are mathematically designed for uncertainty; they provide a built-in measure of how much the model "knows" based on the distance between the test point and the training data.

Massimiliano Patacchiola and Amos Storkey demonstrated this with Deep Kernel Transfer (DKT), which combines neural feature extractors with GPs. This approach is especially powerful in specialized fields like drug discovery, where Henry Moss and Ryan-Rhys Griffiths used it to guide active learning by identifying which molecules the model was most uncertain about.

Amortizing Inference for Efficiency

The primary drawback of Bayesian methods is computational cost. To solve this, researchers use Amortized Inference, where a secondary network is trained to "predict" the posterior distribution of the first network, bypassing expensive optimization steps.

Zhuo Sun and Jing-Hao Xue applied this to prototypes, treating class centers as random variables. Similarly, Matthew Ashman and Adrian Weller developed A-POVI-BNN, which meta-learns how to perform Bayesian inference on task-specific networks, providing well-calibrated predictions even when data is extremely scarce.

Statistical Bias and Selective Prediction

Sometimes, the lack of calibration isn't just a Bayesian problem—it's a statistical one. Saba Ghaffari and Yu-Xiong Wang showed that standard classifiers are statistically biased when trained on few examples. By applying Firth bias reduction, they consistently improved both accuracy and calibration across various benchmarks.

When calibration is still difficult to achieve, models can use selective prediction (or abstention). Yanda Chen and He He introduced SENSEL, a method that measures how sensitive a model's prediction is to minor prompt changes. If a prediction "flips" easily, the model abstains from answering, effectively knowing when it is likely to be wrong.

By moving away from simple point estimates and addressing the inherent biases of low-data regimes, these researchers are building few-shot systems that are not just smart, but self-aware.

Papers in this version

Showing 60 of 90 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_uncertainty_calibration_for_few_shot_lea_3322c17b,
  title = {Uncertainty Calibration for Few-Shot Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/uncertainty-calibration-for-few-shot-learning/rdr_3322c17b1ca446c9ae75f6d644bf822d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_uncertainty_calibration_for_few_shot_lea_3322c17b_v1,
  title = {Uncertainty Calibration for Few-Shot Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/uncertainty-calibration-for-few-shot-learning/rdr_3322c17b1ca446c9ae75f6d644bf822d/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}