Improving Task-Specific Generalization in Few-Shot Learning via Adaptive Vicinal Risk Minimization

Long-Kai Huang, Ying WeiNeurIPS 2022 · OpenReview

Overview

Few-shot learning (FSL) models often overfit because a tiny training set—known as the support set—rarely represents the true underlying data distribution accurately. This paper introduces Adaptive Vicinal Risk Minimization (ADV), a method that leverages unlabeled data and a random walk algorithm to estimate a more accurate, adaptive "neighborhood" (vicinal distribution) around each training sample. By doing so, ADV significantly boosts a model's ability to generalize to unseen data.

The Challenge of Task-Specific Generalization

In few-shot learning, models are trained to adapt to novel tasks using only a handful of labeled examples. While most research focuses on meta-generalization (learning a robust feature extractor across many different tasks), task-specific generalization is frequently overlooked.

When a model only has 1 to 5 samples per class, the empirical distribution of the training data is almost certainly biased and deviates from the true population distribution. Training a classifier by simply minimizing the empirical loss on these few samples inevitably leads to overfitting.

Comparison of decision boundaries: (a) Empirical Risk Minimization, (b) standard Vicinal Risk Minimization, and (c) Adaptive Vicinal Risk Minimization.

Techniques like Vicinal Risk Minimization (VRM)—most famously implemented as Mixup—help in standard deep learning by smoothing the space between data points. However, they fall short in FSL. Interpolating between a few highly biased samples merely creates more biased samples. The authors tackle this by estimating a more accurate distribution around each support sample to prevent overfitting.

Proposed Method: ADaptive Vicinal Few-Shot Learning (ADV)

To prevent overfitting to the biased support set, the authors propose ADV. This approach operates during the meta-testing phase (when the model adapts to a new task) and consists of two main technical innovations:

1. Adaptive Vicinal Distributions via Lazy Random Walks

Instead of assuming the true class center lies exactly on the provided labeled sample, ADV uses unlabeled data (specifically, the query set, making this a transductive learning method). It constructs a transition probability matrix based on feature similarities and performs a "lazy random walk" starting from each labeled support sample. By observing which unlabeled samples the walk visits, the algorithm calculates a weighted mean and variance. This creates a Gaussian "vicinity" around the sample that is pulled closer to the true class center, rather than being anchored to an outlier.

2. Expected Vicinal Loss Derivation

Instead of using computationally expensive Monte Carlo sampling to draw data points from these new Gaussian distributions, the authors mathematically derive the expected loss over these distributions.

  • Cross-Entropy (CE) Classifier: They use a second-order Taylor expansion to approximate the expected loss.
  • Support Vector Machine (SVM): They derive a closed-form objective for a multi-class kernel SVM that explicitly incorporates the variance of the vicinal distributions.

Key Insights

  • Mixup isn't enough for Few-Shot Learning: Standard Mixup applied to the support set provides negligible improvements (and sometimes hurts performance) in FSL. Because the support set is so small, the convex combinations generated by Mixup fail to cover the true distribution.
  • The "Center" of a sample isn't the sample itself: By allowing the mean of the vicinal distribution to shift away from the original labeled sample and toward a cluster of similar unlabeled samples, the model avoids memorizing outliers.
  • Variance matters: An ablation study showed that just shifting the mean of the distribution improves performance, but explicitly incorporating the variance (the spread of the random walk paths) into the loss function provides an additional, crucial boost to prevent overfitting.

Key Results

The authors evaluated their method on three standard few-shot image classification benchmarks: miniImageNet, CUB (fine-grained birds), and CIFAR-FS. ADV achieves state-of-the-art or highly competitive results across the board, particularly shining in the ultra-scarce 1-shot setting.

Performance comparison of ADV against state-of-the-art few-shot learning baselines on miniImageNet, CUB, and CIFAR-FS benchmarks.
  • miniImageNet: ADV-CE achieved 74.63% accuracy on 5-way 1-shot tasks, outperforming strong transductive baselines like SIB (70.0%) and FreeLunch (68.57%).
  • CUB & CIFAR-FS: ADV-CE reached 84.20% (1-shot CUB) and 82.02% (1-shot CIFAR-FS), consistently beating baselines by 2-4%.
  • Architecture Agnostic: The performance boosts were consistent regardless of the underlying feature extractor (tested on a shallow 4-layer CNN, ResNet-12, and WideResNet-28).
  • Synergy with other methods: When combined with Transductive Information Maximization (TIM), ADV-CE achieved 80.75% on 1-shot miniImageNet, competing with the absolute state-of-the-art in transductive FSL.

Limitations and Future Work

  • Approximation Errors: Because the Cross-Entropy loss cannot be computed in a closed form over a Gaussian distribution, the authors rely on a second-order Taylor expansion. This approximation error can potentially lead to sub-optimal performance. Deriving a tighter approximation is a key direction for future work.
  • Loss Function Compatibility: Currently, the mathematical derivations for ADV are strictly limited to Cross-Entropy and SVM classifiers. Extending this framework to other loss functions requires custom mathematical derivations.
  • Transductive Reliance: The method requires access to a batch of unlabeled data (the query set) at inference time to perform the random walk, which may not be feasible in strict online or streaming prediction scenarios where queries arrive one by one.

Summary updated

Abstract

Recent years have witnessed the rapid development of meta-learning in improving the meta generalization over tasks in few-shot learning. However, the task-specific level generalization is overlooked in most algorithms. For a novel few-shot learning task where the empirical distribution likely deviates from the true distribution, the model obtained via minimizing the empirical loss can hardly generalize to unseen data. A viable solution to improving the generalization comes as a more accurate approximation of the true distribution; that is, admitting a Gaussian-like vicinal distribution for each of the limited training samples. Thereupon we derive the resulting vicinal loss function over vicinities of all training samples and minimize it instead of the conventional empirical loss over training samples only, favorably free from the exhaustive sampling of all vicinal samples. It remains challenging to obtain the statistical parameters of the vicinal distribution for each sample. To tackle this challenge, we further propose to estimate the statistical parameters as the weighted mean and variance of a set of unlabeled data it passed by a random walk starting from training samples. To verify the performance of the proposed method, we conduct experiments on four standard few-shot learning benchmarks and consolidate the superiority of the proposed method over state-of-the-art few-shot learning baselines.
BibTeX
@inproceedings{huang2022improving,
  author = {Long-Kai Huang and Ying Wei},
  title = {{Improving Task-Specific Generalization in Few-Shot Learning via Adaptive Vicinal Risk Minimization}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=fHUBa3gQno},
}
Figure