Improve the Performance of Transfer Learning Without Fine-Tuning Using Dissimilarity-Based Multi-view Learning for Breast Cancer Histology Images

Simon Bernard, Hongliu Cao, Simon Bernard 0001, Laurent Heutte, Robert Sabourin

TL;DR

This paper demonstrates that you don’t always need to fine-tune massive deep learning models to get state-of-the-art results in medical imaging. By using pre-trained networks as "off-the-shelf" feature extractors and fusing them with traditional handcrafted features through a clever dissimilarity-based multi-view learning approach, the authors achieved a peak accuracy of 93% on breast cancer histology classification.

What Problem Does This Paper Address?

Deep learning has revolutionized image classification, but it has a "hunger" problem: it requires massive amounts of labeled data. In medical fields like histology (the study of microscopic tissue structure), datasets are often tiny. This paper tackles the High Dimensional Low Sample Size (HDLSS) problem.

When you only have a few hundred images, training a deep neural network from scratch is impossible, and even fine-tuning can lead to overfitting. Furthermore, researchers often struggle to combine different types of data—like "handcrafted" texture features and deep learning features—without the resulting feature vector becoming so large that it overwhelms the small number of available samples.

Methods

The authors propose a pipeline that treats different feature sets as different "views" of the same data:

  1. Feature Extraction (No Fine-Tuning): Instead of training, they use five deep learning models (ResNet-18, ResNet-152, ResNeXt, NASNet-A, and VGG16) pre-trained on ImageNet. They also extract traditional handcrafted features: PFTAS (morphological statistics) and GLCM (texture matrices).
  2. Random Forest Dissimilarity (RFD): For each feature group (or "view"), they train a Random Forest. They calculate a "dissimilarity" score between images based on how often they end up in different leaf nodes of the trees. If two images consistently land in the same leaf, they are considered similar.
  3. Multi-View Fusion (RFSVM): They average the dissimilarity matrices from all six views to create a single, joint similarity matrix.
  4. Classification: This joint matrix is used as a custom kernel for a Support Vector Machine (SVM). This approach, called RFSVM, effectively reduces the high-dimensional feature space into a manageable similarity-based representation.

Datasets

The study uses the ICIAR 2018 Grand Challenge on Breast Cancer Histology Images dataset.

  • Composition: 400 H&E (Hematoxylin and eosin) stained microscopy images.
  • Classes: The images are balanced across four categories: Normal, Benign, In situ carcinoma, and Invasive carcinoma.
  • Protocol: The authors used a 75/25 train-test split, repeated 10 times to ensure statistical robustness.

Key Insights and Ideas

  • Deep Learning Beats Handcrafted (Mostly): Even though the deep networks were trained on generic objects (like cats and cars in ImageNet), they significantly outperformed specialized handcrafted features for histology when used as feature extractors. ResNet-152 was the top individual performer.
  • The Power of "Weak" Features: On its own, the handcrafted feature set was the worst performer (67% accuracy). However, when added to the deep learning ensemble, it boosted the overall accuracy. This proves that handcrafted features contain complementary information that deep networks might miss.
  • Dissimilarity as a Dimensionality Reducer: By converting high-dimensional features (up to 31,855 dimensions) into a dissimilarity matrix (400x400), the authors bypassed the "curse of dimensionality" that usually plagues small datasets.

Key Results

The integration of multiple "views" showed a clear upward trend in performance:

  • Handcrafted Features Only: 67.00% average accuracy.
  • Best Single DL Model (ResNet-152): 79.30% average accuracy.
  • Combining 5 DL Models (RFSVM-DL): 82.90% average accuracy.
  • The Full Ensemble (RFSVM-All): 87.10% average accuracy, with a "best run" hitting 93.00%.

The model showed particularly high sensitivity (96%) for detecting invasive carcinoma, which is critical for clinical applications.

Limitations and Future Work

While the results are impressive, the authors note a few areas for improvement:

  • Patch-level Information: To simplify the process, the authors rescaled whole images to fit the networks. Other state-of-the-art methods use "patch-wise" training (breaking images into smaller pieces), which might capture finer details.
  • Computational Cost: Running six different feature extractors and building multiple Random Forests is computationally heavier than using a single model.
  • Dataset Size: While 400 images is standard for this challenge, the authors suggest that testing on even larger, more diverse datasets would be necessary to validate the "multi-view" approach for clinical use.

Future work could involve exploring different ways to weight the "views" (e.g., giving more importance to ResNet-152 than the handcrafted features) rather than simple averaging.

Summary updated

Abstract

Breast cancer is one of the most common types of cancer and leading cancer-related death causes for women. In the context of ICIAR 2018 Grand Challenge on Breast Cancer Histology Images, we compare one handcrafted feature extractor and five transfer learning feature extractors based on deep learning. We find out that the deep learning networks pretrained on ImageNet have better performance than the popular handcrafted features used for breast cancer histology images. The best feature extractor achieves an average accuracy of 79.30%. To improve the classification performance, a random forest dissimilarity based integration method is used to combine different feature groups together. When the five deep learning feature groups are combined, the average accuracy is improved to 82.90% (best accuracy 85.00%). When handcrafted features are combined with the five deep learning feature groups, the average accuracy is improved to 87.10% (best accuracy 93.00%).
BibTeX
@inproceedings{bernard2018improve,
  author = {Simon Bernard and Hongliu Cao and Simon Bernard and Laurent Heutte and Robert Sabourin},
  title = {{Improve the Performance of Transfer Learning Without Fine-Tuning Using Dissimilarity-Based Multi-view Learning for Breast Cancer Histology Images}},
  booktitle = {Iciar},
  year = {2018},
  doi = {10.1007/978-3-319-93000-8\_88},
}

Loading related content...

Figure