The Fragility of Small Sample Sizes
In standard deep learning, large datasets provide a form of statistical insurance. If 5% of the labels in a dataset of 10,000 images are corrupted, an optimization algorithm like Stochastic Gradient Descent will easily average out the noise. The sheer volume of correct examples overpowers the occasional erroneous gradient update, allowing the model to learn a stable, generalized decision boundary.
In few-shot learning, this insurance policy vanishes. When a model must learn a new category from only five examples (a 5-shot scenario), each individual sample carries immense statistical weight, representing 20% of the entire training signal for that class. If even a single sample is mislabeled or highly unrepresentative, it does not merely introduce a slight bias—it fundamentally warps the model's understanding of the category.
As noted by Nikhil Mehta et al. in HyperMix: Out-of-Distribution Detection and Classification in Few-Shot Settings, few-shot models are uniquely vulnerable because every single support sample directly dictates the resulting decision boundary. This lack of statistical redundancy means that standard deep learning architectures and denoising techniques overfit to the noise, mistaking corrupted features or incorrect labels for legitimate class characteristics.
How Noise Poisons Class Representations
To understand why label noise is so destructive in low-data regimes, we must look at how few-shot models construct class representations. Many state-of-the-art few-shot frameworks rely on metric learning or prototype-based inference. The core idea is simple: project samples into a high-dimensional embedding space, compute an average representation (a "prototype") for each class, and classify new query instances based on their proximity to these prototypes.
Clean 5-shot Support Set:
[ x ] [ x ] [ x ] [ x ] [ x ] ---> ( True Prototype Center )
Poisoned 5-shot Support Set (1 Mislabeled/Outlier):
[ x ] [ x ] [ x ] [ x ] [ NOISE ] ------------------------> ( Poisoned Prototype Center )
When the support set is clean, the prototype sits at the geometric center of the class cluster. However, as explained by Liang Zhang et al. in A Self-Denoising Model for Robust Few-Shot Relation Extraction, averaging embeddings makes prototypes highly sensitive to outliers. A single mislabeled instance acts as a powerful geometric anchor, pulling the prototype away from the true cluster center and into parts of the embedding space belonging to opposing classes.
This problem compounds in graph- or relation-based few-shot models. Yaqing Wang et al. demonstrate in Property-Aware Relation Networks for Few-Shot Molecular Property Prediction that similarity graphs become highly distorted when a single neighbor from an opposing class is introduced into a small support set. Because the model relies on local graph topology to propagate labels, a single misaligned node degrades the prediction accuracy of the entire local neighborhood.
The sensitivity is not limited to computer vision or graph networks; it extends to natural language tasks as well. In prompt-based fine-tuning for text classification, Cornelia Caragea et al. show in Feature Normalization and Cartography-Based Demonstrations for Prompt-Based Fine-Tuning on Emotion-Related Tasks that few-shot models exhibit extreme sensitivity to noise in both input text and ground-truth labels, leading to highly unstable decision boundaries when human annotators disagree on subjective categories. Similarly, in structured text generation, Zhiyu Chen et al. find in Few-Shot NLG with Pre-Trained Language Model that target descriptions containing information absent from the source data cause few-shot models to quickly learn incorrect associations and hallucinate facts, a degradation that happens far more rapidly than in large-scale training.
The Cascading Errors of Pseudo-Labeling
The vulnerability to label noise becomes even more pronounced when few-shot learning is extended to semi-supervised or self-distillation frameworks. In these setups, a model uses a tiny pool of labeled "seed" data to predict labels for a massive pool of unlabeled data (pseudo-labeling), and then trains itself on its own predictions.
This creates a dangerous feedback loop known as the distillation dilemma. As identified by Philippe Weinzaepfel et al. in Barely-Supervised Learning: Semi-supervised Learning with Very Few Labeled Images, semi-supervised learning in extreme low-data regimes (such as 4 to 8 images per class) is hyper-sensitive to the accuracy of the initial seeds. If one seed label is incorrect, the model will generate low-quality pseudo-labels for the unlabeled pool.
Instead of correcting the error, the model propagates it exponentially. Trung Bui et al. explore this boundary in A Joint Learning Approach based on Self-Distillation for Keyphrase Extraction from Scientific Documents, noting that below a certain threshold of clean labeled instances, a teacher model's pseudo-labels become so noisy that they completely derail the student model's learning process, leading to catastrophic divergence.
Strategies for Robust Few-Shot Adaptation
To make few-shot models viable in real-world scenarios where data is inherently messy, researchers have developed specialized mechanisms designed to bypass or correct support set noise.
1. Iterative Soft Clustering and Self-Denoising
Rather than blindly averaging support embeddings, robust models treat prototype formation as an optimization problem. For instance, Pratik Mazumder et al. introduce RNNP (RNNP: A Robust Few-Shot Learning Approach), which generates synthetic hybrid features and uses an iterative soft k-means clustering process during inference. By assigning soft weights to support samples, the model dynamically downweights outliers, preventing them from pulling the prototype center away from the true data distribution.
2. Decoupled and Global Data Augmentation
Traditional data augmentation techniques like Mixup can inadvertently worsen label ambiguity if they mix samples locally without understanding the broader data structure. To solve this, Xiangjin Xie proposes a framework called Global Mixup (Global Mixup: Eliminating Ambiguity with Clustering), which decouples sample generation from labeling. By using global clustering relationships to assign labels to synthetic data, it ensures that the generated samples smooth out decision boundaries rather than introducing further noise into data-scarce environments.
3. Cross-Verification via Foundation Models
When human annotations are highly unreliable, some frameworks leverage the collective knowledge of pre-trained foundation models to cross-examine the support set. Shuai Shao et al. introduce the CO3 framework (Collaborative Consortium of Foundation Models for Open-World Few-Shot Learning), which orchestrates an ensemble of distinct foundation models (such as CLIP, DINO, and GPT-3). By validating support samples across multi-modal representations, the system can identify, isolate, and correct noisy labels even when a significant portion of the initial training signal is corrupted.
Key Foundational Work
Accuracy of distance metric learning algorithms (2009) — contributed 2 concept elements to this direction.
Adding Unlabeled Samples to Categories by Learned Attributes (2013) — contributed 1 concept elements to this direction.