Abstract
Deep learning classifiers typically struggle with severe overfitting when trained on extremely small datasets. This paper introduces Classification via Segmentation (CvS), a novel framework that derives classification labels by first predicting segmentation maps. By utilizing a simple label propagation technique to mitigate the high cost of manual segmentation, CvS significantly outperforms standard classifiers in scenarios where only 1 to 100 examples per class are available.
Introduction
Modern deep learning success is often predicated on the availability of massive datasets. However, in specialized fields such as medical imaging or rare disease diagnosis, data is inherently scarce. When forced to train on just a handful of examples (e.g., 1-5 per class), standard classification networks severely overfit. While Multi-Task Learning (MTL) can provide regularization by combining classification and segmentation heads, it often suffers from loss-balancing issues where one task dominates the other. CvS addresses these challenges by injecting dense supervisory signals from segmentation into a single-headed classification framework without the complexities of MTL.
Methodology
Unlike traditional Multi-Task Learning, which employs separate heads for classification and segmentation, CvS is a single-headed approach. The network extracts features and predicts a segmentation map (including classes and background). It then applies a pooling or averaging function over these maps, followed by a softmax layer, to output the final classification label. In this way, the classification is a direct mathematical derivative of the segmentation.
To overcome the manual labeling bottleneck, the authors employ pseudo-mask generation techniques:
- Binarization: For simple datasets like MNIST, pixel values are thresholded to create segmentation maps.
- Label Propagation: For complex natural images, a tiny subset (e.g., 10 images per class) is manually segmented to train a preliminary model. This model then predicts pseudo-segmentation labels for the rest of the training data, which are used to train the main CvS model.
Experimental Results
The efficacy of CvS was evaluated across several benchmarks, demonstrating superior performance in low-data regimes:
- CIFAR-10: With only 1 sample per class, CvS achieved 39.31% accuracy, more than doubling the 18.85% achieved by a standard ResNet101 classifier. With 10 samples per class, CvS reached 73.94% accuracy compared to 26.01% for the standard baseline.
- Medical Imaging (HRF): On the 45-image High Resolution Fundus Photographs dataset, CvS achieved 82.22% accuracy, outperforming both standard classification (66.67%) and MTL (70.23%).
Discussion and Key Insights
Forcing a network to predict pixel-wise segmentation maps acts as a powerful structural prior. It prevents the network from memorizing arbitrary background noise and forces it to learn the actual shape of the object, which is crucial when data is scarce. Furthermore, by deriving classification directly from the segmentation map, CvS completely bypasses the loss-balancing issues that plague traditional multi-task learning.
A critical finding is the cost-efficiency of dense labeling. The authors' analysis reveals that spending time manually segmenting just 10 images per class yields higher classification accuracy than spending that same time applying standard class labels to hundreds of images.
Conclusion
CvS proves that investing in a few dense labels is a more efficient use of human labeling budgets than gathering many sparse labels in low-data scenarios. While it does not yet outperform massive pre-trained models like Big Transfer (BiT) on general datasets, its performance in specialized domains like medical imaging—where pre-training often fails to transfer—is significant. Future work will explore integrating more advanced semi-supervised techniques during the label propagation step to further enhance performance.
Summary updated