Unravelling Small Sample Size Problems in the Deep Learning World

Rohit Keshari, Soumyadeep Ghosh, Saheb Chhabra, Mayank Vatsa, Richa Singh

Abstract

The growth and success of deep learning approaches can be attributed to two major factors: availability of hardware resources and availability of large number of training samples. For problems with large training databases, deep learning models have achieved superlative performances. However, there are a lot of small sample size or $S^3$ problems for which it is not feasible to collect large training databases. It has been observed that deep learning models do not generalize well on $S^3$ problems and specialized solutions are required. In this paper, we first present a review of deep learning algorithms for small sample size problems in which the algorithms are segregated according to the space in which they operate, i.e. input space, model space, and feature space. Secondly, we present Dynamic Attention Pooling approach which focuses on extracting global information from the most discriminative sub-part of the feature map. The performance of the proposed dynamic attention pooling is analyzed with state-of-the-art ResNet model on relatively small publicly available datasets such as SVHN, C10, C100, and TinyImageNet.

Introduction: The Challenge of Data Scarcity

Deep learning typically thrives on massive datasets, but real-world scenarios like rare disease diagnosis or niche image captioning often suffer from a "Small Sample Size" ($S^3$) problem. When training data is insufficient to learn high-dimensional features, models fail to generalize, leading to inaccurate outputs and severe overfitting. This paper addresses the Small Sample Learning (SSL) challenge by organizing the fragmented landscape of SSL research and proposing a structural architectural change to help standard models perform better when data is limited.

A Taxonomy of Small Sample Learning

The authors categorize SSL algorithms based on where they operate in the deep learning pipeline: input space, model space, or feature space.

Taxonomy of Small Sample Learning techniques across Input, Model, and Feature spaces.
  • Input Space: Focuses on data augmentation, domain adaptation, and "Data Fine-tuning" (DFT), which learns to perturb input data to better fit a pre-trained model's decision boundary.
  • Model Space: Includes traditional fine-tuning (MFT), knowledge distillation (Teacher-Student networks), and specialized regularization techniques like guided dropout.
  • Feature Space: Utilizes Deep Metric Learning (DML) to minimize intra-class distance and maximize inter-class distance, making features more discriminative even with fewer samples.

Dynamic Attention Pooling (DAP)

Standard Convolutional Neural Networks (CNNs) typically use Global Average Pooling (GAP) at the final layer. The authors argue that GAP can be detrimental in small-sample regimes because it averages the entire feature map, potentially "washing out" important signals with noise or sparse values.

Comparison between Global Pooling (a) and the proposed Dynamic Attention Pooling (b).

DAP replaces GAP with a more surgical approach. It applies local average pooling using a small sliding window across the feature map. Each resulting local feature vector is fed into a separate softmax layer. The model learns weights for each local region, effectively "attending" to the most useful sub-parts of the image. By selectively dropping insignificant feature map regions, the model improves its ability to generalize to unseen test data.

Experimental Evaluation

The proposed DAP method was evaluated using a ResNet18 backbone on four benchmarks: CIFAR-10 (C10), CIFAR-100 (C100), SVHN, and TinyImageNet.

Key Results

DAP consistently outperformed standard pooling methods (GAP and Global Max Pooling) across all datasets:

  • CIFAR-10: Achieved 95.53% accuracy (vs. 93.78% for GAP).
  • TinyImageNet: Achieved 63.94% accuracy (vs. 61.96% for GAP).
  • CIFAR-100: Showed a marginal but steady improvement to 77.58% (vs. 77.01% for GAP).
  • SVHN: Matched state-of-the-art performance at 96.52%.

Conclusion

The research highlights that "where" an optimization occurs matters. By classifying SSL techniques, the authors provide a framework for combining strategies. Furthermore, the insight that GAP might be a bottleneck for small datasets is significant. Providing "proper pooling" through attention-based mechanisms like DAP can significantly improve the generalization of deep models when data is scarce. Future work may explore computational overhead on larger architectures and performance in extreme scarcity scenarios.

Summary updated

Abstract

The growth and success of deep learning approaches can be attributed to two major factors: availability of hardware resources and availability of large number of training samples. For problems with large training databases, deep learning models have achieved superlative performances. However, there are a lot of \textit{small sample size or $S^3$} problems for which it is not feasible to collect large training databases. It has been observed that deep learning models do not generalize well on $S^3$ problems and specialized solutions are required. In this paper, we first present a review of deep learning algorithms for small sample size problems in which the algorithms are segregated according to the space in which they operate, i.e. input space, model space, and feature space. Secondly, we present Dynamic Attention Pooling approach which focuses on extracting global information from the most discriminative sub-part of the feature map. The performance of the proposed dynamic attention pooling is analyzed with state-of-the-art ResNet model on relatively small publicly available datasets such as SVHN, C10, C100, and TinyImageNet.
BibTeX
@inproceedings{keshari2020unravelling,
  author = {Rohit Keshari and Soumyadeep Ghosh and Saheb Chhabra and Mayank Vatsa and Richa Singh},
  title = {{Unravelling Small Sample Size Problems in the Deep Learning World}},
  booktitle = {BigMM},
  year = {2020},
  url = {https://openreview.net/forum?id=f1m17oc1hB_9},
}
Figure