Few-Shot Learning via Feature Hallucination with Variational Inference

Qinxuan Luo, Lingfeng Wang, Jingguo Lv, Shiming Xiang, Chunhong Pan

Abstract

Few-shot learning (FSL) aims to adapt deep learning models to new categories using only a handful of labeled examples, a task often hampered by the risk of overfitting. This paper introduces VI-Net, a generative framework that addresses data scarcity by "hallucinating" synthetic feature vectors. By employing Variational Inference to map limited support samples into a structured latent distribution, VI-Net generates diverse features that sharpen classifier decision boundaries. Experimental results on mini-ImageNet and CUB benchmarks demonstrate that VI-Net achieves state-of-the-art performance among augmentation-based methods.

Introduction

Deep learning models typically require vast amounts of labeled data to generalize effectively. In Few-Shot Learning (FSL), models must adapt using only a small "support set" (e.g., 1 or 5 samples). The primary obstacle is the high risk of overfitting, as sparse data makes it difficult for classifiers to establish robust decision boundaries for unseen "query" samples.

While data augmentation is a standard remedy, traditional methods often rely on transferring intra-class variations (such as pose or lighting) from base categories. This approach can be imprecise; slight misalignments in the transfer direction can lead to generated samples that drift from the true class center, potentially degrading model performance.

Methodology: The VI-Net Framework

The proposed VI-Net architecture consists of four primary modules: a feature extractor ($F$), a distribution encoder ($E$), a feature decoder ($D$), and a cosine-similarity classifier ($C$).

Overall architecture of VI-Net, illustrating the flow from feature extraction to feature hallucination and classification.

Training and Inference Workflow

The framework operates in two distinct stages:

  1. Pre-training: A feature extractor and cosine classifier are trained on a large "base" dataset to establish general visual representations.
  2. Base-training (Generative Module): With the feature extractor frozen, the encoder $E$ learns to map features into a low-dimensional latent space. Rather than simple point mapping, $E$ estimates the parameters ($\mu, \sigma^2$) of a Gaussian distribution for each category.

Variational Inference and Hallucination

To ensure the latent space is representative, the model employs a KL-divergence loss ($L_{lat}$). This aligns the "posterior" distribution, derived from the support samples, with a "prior" distribution learned from the broader training set.

Detailed view of the distribution estimation process using Variational Inference to align posterior and prior distributions.

Once the distribution is established, the model samples random vectors from this space and passes them through the decoder $D$ to "hallucinate" new feature vectors ($\tilde{f}$). During inference, these synthetic features augment the original support samples to fine-tune the cosine classifier, effectively filling gaps in the feature space.

Experimental Evaluation

VI-Net was evaluated on two benchmark datasets: mini-ImageNet (general object recognition) and CUB (fine-grained bird species classification).

Key Results

  • mini-ImageNet: VI-Net achieved 61.05% accuracy in 1-shot tasks and 78.60% in 5-shot tasks, outperforming baseline classifiers and competing augmentation methods like Delta-Encoder.
  • CUB: Performance was notably strong, with 74.76% (1-shot) and 86.84% (5-shot) accuracy. The success on fine-grained data suggests that Variational Inference is particularly effective when intra-class differences are subtle.
  • Ablation Studies: The necessity of the three loss functions—classification ($L_{cla}$), reconstruction ($L_{rec}$), and latent constraint ($L_{lat}$)—was confirmed. Removing $L_{lat}$ led to a significant accuracy drop, as generated features lacked the diversity needed to generalize.

Discussion and Key Insights

VI-Net's effectiveness stems from its focus on feature-level hallucination rather than image generation. This approach is computationally efficient and directly targets the classifier's requirements. By treating categories as distributions, the model generates "diverse but plausible" variations, which sharpens the decision boundary and provides the classifier with a more comprehensive view of the class manifold.

Limitations and Future Work

While VI-Net performs well, 1-shot scenarios remain challenging, as estimating a reliable distribution from a single data point is inherently difficult. Future research may explore incorporating more sophisticated prior knowledge from base classes to assist in 1-shot distribution estimation, potentially by learning a generalized "variance map" applicable across similar categories.

Summary updated

Abstract

Deep learning has achieved huge success in the field of artificial intelligence, but the performance heavily depends on labeled data. Few-shot learning aims to make a model rapidly adapt to unseen classes with few labeled samples after training on a base dataset, and this is useful for tasks lacking labeled data such as medical image processing. Considering that the core problem of few-shot learning is the lack of samples, a straightforward solution to this issue is data augmentation. This paper proposes a generative model (VI-Net) based on a cosine-classifier baseline. Specifically, we construct a framework to learn to define a generating space for each category in the latent space based on few support samples. In this way, new feature vectors can be generated to help make the decision boundary of classifier sharper during the fine-tuning process. To evaluate the effectiveness of our proposed approach, we perform comparative experiments and ablation studies on mini-ImageNet and CUB. Experimental results show that VI-Net does improve performance compared with the baseline and obtains the state-of-the-art result among other augmentation-based methods.
BibTeX
@inproceedings{luo2021few,
  author = {Qinxuan Luo and Lingfeng Wang and Jingguo Lv and Shiming Xiang and Chunhong Pan},
  title = {{Few-Shot Learning via Feature Hallucination with Variational Inference}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  doi = {10.1109/wacv48630.2021.00401},
}
Figure