On evaluating CNN representations for low resource medical image classification

Taruna Agrawal, Rahul Gupta, Shrikanth Narayanan

TL;DR

Transfer learning is the go-to strategy for medical image classification when data is scarce, but choosing the right pre-trained model often feels like guesswork. This paper introduces a "one-shot" metric that uses t-SNE projections and Gaussian modeling on the training set to predict which CNN architecture will perform best on the test set, achieving an 87% correlation with actual results.

What Problem Does This Paper Address?

Deep learning, specifically Convolutional Neural Networks (CNNs), has transformed image recognition. However, these models are data-hungry. In medical imaging, we often face "low-resource" scenarios where we only have a few thousand labeled images—far too few to train a massive CNN from scratch.

The standard solution is transfer learning: take a model pre-trained on a massive general dataset (like ImageNet) and "fine-tune" it or use it as a feature extractor for the medical task. But here’s the catch: there are dozens of popular CNN architectures (VGG, ResNet, Inception, etc.). Which one will work best for a specific medical dataset? Currently, researchers usually have to train and test every single one to find out, which is computationally expensive and time-consuming. The authors aim to find a way to predict a model's performance using only the training data.

Methods

The researchers compared two main approaches for gastrointestinal (GI) landmark classification:

  1. Baseline "Kitchen Sink": An assembly of traditional, hand-crafted "knowledge-driven" features like color layouts, texture descriptors (Tamura features), and edge histograms.
  2. CNN Transfer Learning: Extracting "data-driven" features from the penultimate layer of five different CNN architectures (VGGNet, ResNet50, Inception-V3, XceptionNet, and MobileNet) pre-trained on ImageNet.

To solve the model selection problem, they proposed Algorithm 1, a novel metric ($A$):

  • Dimensionality Reduction: High-dimensional CNN features are projected into a 2D space using t-SNE.
  • Probabilistic Modeling: Each class in this 2D space is modeled as a Gaussian distribution.
  • Confusion Estimation: They calculate the probability that a data point sampled from a specific class distribution actually belongs to that class (i.e., it doesn't overlap too much with other classes).
  • Final Metric: The average of these probabilities across all classes gives the metric $A$.

Datasets

The study uses the KVASIR dataset, which contains 8,000 images of the gastrointestinal tract. These images are equally divided into eight classes representing different anatomical landmarks, pathological findings, or clinical procedures (e.g., esophagitis, polyps, ulcerative colitis). The authors used a standard split of 4,000 images for training and 4,000 for testing.

Key Insights and Ideas

  • Data-Driven > Hand-Crafted: In almost every case, features learned by CNNs on general images outperformed hand-crafted medical imaging features. This suggests that the geometric patterns CNNs learn are highly transferable, even to specialized medical domains.
  • The Power of t-SNE for Modeling: While t-SNE is usually just a visualization tool, the authors found that its 2D projections were robust enough to allow for accurate probability modeling, even with limited data. Other methods like PCA resulted in too much class overlap to be useful for their metric.
  • Model Agnosticism: The proposed metric $A$ doesn't need to know what kind of final classifier (like an SVM) you are using. It evaluates the "discriminative capability" of the feature representation itself.
  • Complementary Features: Combining all CNN and baseline features yielded the highest accuracy, suggesting that different architectures "see" different useful aspects of the medical images.

Key Results

  • Performance: MobileNet emerged as the best individual performer (81.7% accuracy), while the "Combined" model reached 83.8%. Interestingly, ResNet50 performed significantly worse (61.1%) than the baseline (71.6%), highlighting the danger of blindly choosing a popular architecture.
  • Predictive Power: The proposed metric $A$ showed a correlation of 0.87 with the actual test set accuracies. As seen in Figure 2, while the metric consistently underestimated the absolute accuracy, it perfectly captured the trend across different models.
Fig. 2. Plot comparing the accuracies obtained on the test set against the estimates A obtained on each feature representation.
  • Visual Confirmation: The t-SNE plots in Figure 3 visually explain why some models work better. In high-performing models like MobileNet (d) or XceptionNet (f), classes form distinct clusters. In the poorly performing ResNet (e), the classes are a jumbled mess.
Fig. 3. t-SNE plots obtained using various image feature representations: (a) Baseline features, (b) Inception-V3, (c) VGG-16, (d) MobileNet, (e) ResNet, (f) XceptionNet, (g) All representations combined. We chose a 2-dimensional projection for the ease of visual inspection.

Limitations and Future Work

The authors acknowledge a few areas for improvement:

  • Absolute Accuracy Gap: While the metric $A$ is excellent for ranking models, its absolute value is not a direct predictor of the final percentage accuracy.
  • Class Confusion: The metric was only "mediocre" (20-50% correlation) at predicting exactly which two classes would be confused with each other (e.g., mistaking polyps for dyed-resection-margins).
  • Layer Selection: This study only looked at the penultimate layer. Future work could investigate whether intermediate layers provide better representations for specific medical tasks.
  • Classifier Design: The authors suggest that t-SNE plots could eventually be used to inform the design of the classifier itself, perhaps using a "mixture of experts" approach for classes that overlap heavily.

Summary updated

Abstract

Convolutional Neural Networks (CNNs) have revolutionized performances in several machine learning tasks such as image classification, object tracking, and keyword spotting. However, given that they contain a large number of parameters, their direct applicability into low resource tasks is not straightforward. In this work, we experiment with an application of CNN models to gastrointestinal landmark classification with only a few thousands of training samples through transfer learning. As in a standard transfer learning approach, we train CNNs on a large external corpus, followed by representation extraction for the medical images. Finally, a classifier is trained on these CNN representations. However, given that several variants of CNNs exist, the choice of CNN is not obvious. To address this, we develop a novel metric that can be used to predict test performances, given CNN representations on the training set. Not only we demonstrate the superiority of the CNN based transfer learning approach against an assembly of knowledge driven features, but the proposed metric also carries an 87% correlation with the test set performances as obtained using various CNN representations.
BibTeX
@inproceedings{agrawal2019evaluating,
  author = {Taruna Agrawal and Rahul Gupta and Shrikanth Narayanan},
  title = {{On evaluating CNN representations for low resource medical image classification}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2019},
  doi = {10.1109/icassp.2019.8682397},
}
Figure