Few-Shot Learning in the Presence of Label Noise

This direction is split into other directions: label noise sensitivity (few-shot label sensitivity). Its last version stays readable and citable here.

Version 1 of 167 papers82 conceptsfrozen 2026-10-04

In standard deep learning, we rely on the "law of large numbers" to handle imperfections. If 5% of a million-image dataset is mislabeled, the model effectively treats that noise as a rounding error, averaging it out across thousands of correct gradients. However, as Jiang Lu and colleagues note in their survey, few-shot learning (FSL) is uniquely fragile. When you only have five examples of a "Tiger," a single mislabeled "Tiger cat" represents 20% of your knowledge base.

This research direction focuses on making few-shot models robust to "dirty" data—scenarios where the limited support set provided at test time, or the tasks used during meta-training, contain incorrect, ambiguous, or partial labels.

The Geometry of a Single Mistake

To understand why noise is so destructive in FSL, consider the mechanism of Prototypical Networks. These models classify new images by calculating a "prototype" (the mean vector) of the available support samples for each class. As Ji Zhang explains in the DETA framework, a single mislabeled sample acts as a powerful anchor that pulls the entire class prototype toward the wrong region of the feature space.

In gradient-based meta-learning (like MAML), the problem is even more acute. A noisy label provides a misleading supervision signal that forces the model to "adapt" its weights in a direction that destroys its generalizable features. Krishnateja Killamsetty and the authors of NESTEDMAML argue that standard meta-learning objectives lack the inherent robustness to ignore these signals, leading to catastrophic overfitting on the tiny, corrupted support set.

Strategies for Denoising the Support Set

Researchers have converged on several mechanisms to prevent a few bad apples from spoiling the model:

1. Dynamic Reweighting and Attention

Instead of treating all support samples equally, models like TraNFS use Transformer-based self-attention to compare support samples against one another. If one sample is a "loner" in the feature space compared to its peers in the same class, the attention mechanism downweights its contribution to the prototype. Similarly, DETA filters out both visual noise (cluttered backgrounds) and label noise simultaneously, ensuring only "clean" information informs the task adaptation.

2. Structural Anchors from Foundation Models

A surprising finding by Cheng-En Wu and others is that prompt tuning in vision-language models like CLIP is naturally more robust to noise than traditional fine-tuning. The fixed class-name tokens and pre-trained embeddings act as a "structural anchor." Because the model's core vocabulary is frozen, it is physically harder for a few noisy few-shot examples to "pull" the model's understanding away from its pre-trained common sense.

3. Principal Trajectory Updates

Rather than just looking at the final state of a model after it adapts to a noisy task, Dong Chen proposed Eigen-Reptile (paper). This method updates meta-parameters along the principal direction of task-specific trajectories. By focusing on the dominant direction of learning across multiple steps, the model effectively filters out the erratic, high-frequency "jitter" caused by individual mislabeled samples.

The Challenge of Partial and Ambiguous Labels

In real-world deployment, noise isn't always a "wrong" label; sometimes it is an "incomplete" one. In Few-Shot Partial-Label Learning, an image might be labeled with a set of candidates (e.g., {Dog, Wolf, Coyote}) because the annotator is uncertain.

Yunfeng Zhao and Guoxian Yu demonstrate in their research that standard FSL models fail here because they assume ground-truth perfection. To combat this, frameworks like FsPML-SF (paper) use synthetic feature networks to generate "clean" prototypical features from these ambiguous sets, effectively diluting the impact of the incorrect candidate labels.

Why This Matters for Deployment

The sensitivity to noise is a primary bottleneck for AI in specialized fields. In medical imaging, Julio Silva-Rodríguez notes that adapting models to new organs using only a few scans is highly susceptible to annotation errors. Similarly, in Natural Language Processing, Buru Chang found that In-Context Learning in Large Language Models suffers a sharp decline in stability when prompt examples are mislabeled.

By moving away from the assumption of "perfect data," this research direction is building the "immune system" for few-shot learners, allowing them to function in the messy, uncertain environments of the real world.

Papers in this version

Showing 60 of 67 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_few_shot_learning_in_the_presence_of_lab_9bcb5307,
  title = {Few-Shot Learning in the Presence of Label Noise},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/few-shot-learning-in-the-presence-of-label-noise/rdr_9bcb5307164742898994b4222425e865},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_few_shot_learning_in_the_presence_of_lab_9bcb5307_v1,
  title = {Few-Shot Learning in the Presence of Label Noise},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/few-shot-learning-in-the-presence-of-label-noise/rdr_9bcb5307164742898994b4222425e865/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}