Improve the performance of transfer learning without fine-tuning using dissimilarity-based multi-view learning for breast cancer histology images

Hongliu Cao, Simon Bernard, Laurent Heutte, Robert Sabourin

Abstract

Breast cancer is one of the most common types of cancer and leading cancer-related death causes for women. In the context of ICIAR 2018 Grand Challenge on Breast Cancer Histology Images, we compare one handcrafted feature extractor and five transfer learning feature extractors based on deep learning. We find out that the deep learning networks pretrained on ImageNet have better performance than the popular handcrafted features used for breast cancer histology images. The best feature extractor achieves an average accuracy of 79.30%. To improve the classification performance, a random forest dissimilarity based integration method is used to combine different feature groups together. When the five deep learning feature groups are combined, the average accuracy is improved to 82.90% (best accuracy 85.00%). When handcrafted features are combined with the five deep learning feature groups, the average accuracy is improved to 87.10% (best accuracy 93.00%).

Introduction

Deep learning has revolutionized image classification, but its application in medical imaging is often hindered by the "High Dimensional Low Sample Size" (HDLSS) problem. Histology datasets typically contain a limited number of labeled images, making training from scratch unfeasible and fine-tuning prone to overfitting. Furthermore, integrating diverse feature types—such as traditional handcrafted texture features and deep learning features—without overwhelming the small sample size is a significant challenge. This paper proposes a dissimilarity-based multi-view learning approach to leverage "off-the-shelf" pre-trained networks alongside traditional features.

Methodology

The proposed pipeline treats different feature sets as distinct "views" of the data:

  1. Feature Extraction: Five deep learning models (ResNet-18, ResNet-152, ResNeXt, NASNet-A, and VGG16) pre-trained on ImageNet are used as feature extractors without fine-tuning. Additionally, handcrafted features including PFTAS (morphological statistics) and GLCM (texture matrices) are extracted.
  2. Random Forest Dissimilarity (RFD): A Random Forest is trained for each feature group. Dissimilarity scores between images are calculated based on their co-occurrence in leaf nodes across the forest.
  3. Multi-View Fusion (RFSVM): The dissimilarity matrices from all views are averaged to create a unified joint similarity matrix.
  4. Classification: This joint matrix serves as a custom kernel for a Support Vector Machine (SVM), effectively reducing high-dimensional features into a manageable similarity-based representation.

Experimental Setup

The study utilizes the ICIAR 2018 Grand Challenge on Breast Cancer Histology Images dataset, consisting of 400 H&E stained microscopy images balanced across four classes: Normal, Benign, In situ carcinoma, and Invasive carcinoma. A 75/25 train-test split was repeated 10 times to ensure statistical robustness.

Results and Discussion

The experimental results demonstrate that deep learning models, despite being trained on generic ImageNet data, significantly outperform specialized handcrafted features for histology. ResNet-152 emerged as the strongest individual feature extractor.

Comparison of classification performance across different feature extraction and fusion methods.

The integration of multiple views led to a clear performance boost. While handcrafted features performed poorly in isolation (67.00%), their inclusion in the ensemble provided complementary information that enhanced the deep learning models. The final ensemble, RFSVM-All, achieved an average accuracy of 87.10% and a peak accuracy of 93.00%.

Confusion matrix and performance metrics (sensitivity and specificity) for the RFSVM-All model.

The model demonstrated high clinical relevance, particularly with a 96% sensitivity for detecting invasive carcinoma. The dissimilarity-based approach effectively bypassed the "curse of dimensionality" by converting high-dimensional features into a compact 400x400 matrix.

Conclusion

This research highlights that state-of-the-art results in medical imaging can be achieved without extensive fine-tuning of massive models. By using dissimilarity-based fusion, researchers can effectively combine the strengths of various feature extractors. Future work will explore patch-level analysis and weighted view integration to further refine performance on larger, more diverse datasets.

Summary updated

Abstract

Breast cancer is one of the most common types of cancer and leading cancer-related death causes for women. In the context of ICIAR 2018 Grand Challenge on Breast Cancer Histology Images, we compare one handcrafted feature extractor and five transfer learning feature extractors based on deep learning. We find out that the deep learning networks pretrained on ImageNet have better performance than the popular handcrafted features used for breast cancer histology images. The best feature extractor achieves an average accuracy of 79.30%. To improve the classification performance, a random forest dissimilarity based integration method is used to combine different feature groups together. When the five deep learning feature groups are combined, the average accuracy is improved to 82.90% (best accuracy 85.00%). When handcrafted features are combined with the five deep learning feature groups, the average accuracy is improved to 87.10% (best accuracy 93.00%).
BibTeX
@inproceedings{cao2018improve,
  author = {Hongliu Cao and Simon Bernard and Laurent Heutte and Robert Sabourin},
  title = {{Improve the performance of transfer learning without fine-tuning using dissimilarity-based multi-view learning for breast cancer histology images}},
  booktitle = {Iciar},
  year = {2018},
  doi = {10.1007/978-3-319-93000-8\_88},
}
Figure