Few-shot Classification via Ensemble Learning with Multi-Order Statistics

Sai Yang, Fan Liu, Delong Chen, Jun Zhou

Introduction

Few-shot Classification (FSC) represents a significant challenge in computer vision, aiming to emulate the human ability to recognize novel categories from only a handful of labeled examples. The standard paradigm involves pre-training a feature extractor on a large "base" dataset and subsequently adapting it to "novel" classes. However, this approach often suffers from overfitting to base-class specific features, leading to poor generalization on unseen categories. While ensemble learning—combining multiple models—is a proven strategy for improving generalization, its application in FSC is often hindered by the prohibitive computational and memory costs of running multiple deep neural network backbones.

In this work, we present ELMOS (Ensemble Learning with Multi-Order Statistics), a framework designed to achieve the benefits of ensemble learning within a single backbone. By extracting and fusing 1st, 2nd, and 3rd-order statistical representations, ELMOS achieves state-of-the-art performance while maintaining high computational efficiency.

Theoretical Motivation

A core contribution of this research is the theoretical justification for ensembling in the context of few-shot learning. The authors provide a mathematical proof demonstrating that reducing the ensemble error on base classes directly tightens the error bound on unseen novel classes, provided the domain shift between the two sets is not extreme. This insight shifts the focus toward creating diverse and complementary representations during the pre-training phase to ensure robust transferability.

The ELMOS Framework

Instead of training multiple independent networks, ELMOS utilizes a single shared backbone (e.g., ResNet12) and branches the output into three parallel statistical modules. This approach captures different "views" of the same feature map without adding significant parameter overhead.

Multi-Order Statistics

  1. 1st-Order (Mean): This branch employs standard Global Average Pooling (GAP) to capture the primary spatial features of the image.
  2. 2nd-Order (Covariance): This branch captures the correlations between different feature channels, providing a more nuanced understanding of texture and structural relationships.
  3. 3rd-Order (Cumulant): By calculating higher-order cumulants, this branch captures non-Gaussian distributions within the data, identifying complex patterns that lower-order statistics might miss.

Joint Training and Fusion

To ensure these branches learn complementary features, ELMOS employs a joint training strategy. The model is optimized using a combination of standard Cross-Entropy loss and Supervised Contrastive loss. This dual-loss approach encourages the model to cluster similar categories together in the feature space while maximizing the diversity of the statistical representations. For the final classification task, the features from all three branches are concatenated into a single, rich "fingerprint" of the image.

Experimental Results

ELMOS was evaluated against several competitive benchmarks, including miniImageNet, tieredImageNet, CIFAR-FS, and the fine-grained CUB (Caltech-UCSD Birds) dataset.

Performance Highlights

  • State-of-the-Art Accuracy: On the CUB dataset, ELMOS achieved 85.21% accuracy in 1-shot tasks and 95.02% in 5-shot tasks, outperforming previous leading methods like DeepBDC by over 1%.
  • Generalization: On miniImageNet, ELMOS consistently outperformed top-tier methods such as PAL and IE by margins of 1.0% to 1.4%.
  • Cross-Domain Robustness: In tests where the model was trained on miniImageNet but evaluated on the CUB dataset, ELMOS demonstrated superior robustness. This suggests that multi-order statistics help the model capture more universal visual features that transcend specific dataset biases.

Efficiency and Complementarity

One of the primary advantages of ELMOS is its efficiency. Because the multi-order pooling modules are parameter-free, the framework adds almost zero parameter overhead compared to a standard single-backbone model. This makes it significantly more practical than traditional multi-model ensembles like EASY.

Furthermore, the different statistical orders provide complementary information. While 1st-order statistics capture fine-grained spatial details, 2nd and 3rd-order statistics are smoother and more robust to noise, leading to a more stable representation of the input data.

Conclusion and Future Work

ELMOS demonstrates that multi-order statistics are a powerful tool for improving few-shot classification. By leveraging higher-order information within a single backbone, the method achieves the performance gains of an ensemble with the efficiency of a single model.

Future research may explore the utility of even higher-order statistics (e.g., 4th or 5th order) and investigate the application of these principles to self-supervised few-shot learning, reducing the reliance on large labeled base sets. Additionally, optimizing the memory intensity of covariance and cumulant calculations remains an important path for scaling the method to even higher-dimensional feature maps.

Summary updated

Abstract

Transfer learning has been widely adopted for few-shot classification. Recent studies reveal that obtaining good generalization representation of images on novel classes is the key to improving the few-shot classification accuracy. To address this need, we prove theoretically that leveraging ensemble learning on the base classes can correspondingly reduce the true error in the novel classes. Following this principle, a novel method named Ensemble Learning with Multi-Order Statistics (ELMOS) is proposed in this paper. In this method, after the backbone network, we use multiple branches to create the individual learners in the ensemble learning, with the goal to reduce the storage cost. We then introduce different order statistics pooling in each branch to increase the diversity of the individual learners. The learners are optimized with supervised losses during the pre-training phase. After pre-training, features from different branches are concatenated for classifier evaluation. Extensive experiments demonstrate that each branch can complement the others and our method can produce a state-of-the-art performance on multiple few-shot classification benchmark datasets.
BibTeX
@inproceedings{yang2023few,
  author = {Sai Yang and Fan Liu and Delong Chen and Jun Zhou},
  title = {{Few-shot Classification via Ensemble Learning with Multi-Order Statistics}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.24963/ijcai.2023/181},
}
Figure