Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning?

Nima Tajbakhsh, Jae Y. Shin, Suryakanth R. Gurudu, R. Todd Hurst, Christopher B. Kendall, Michael B. Gotway, Jianming Liang

Abstract

This study addresses a critical question in medical imaging: whether deep learning models should be trained from scratch on domain-specific data or fine-tuned from pre-trained models on natural images. By evaluating four diverse clinical tasks—polyp detection, image quality assessment, pulmonary embolism detection, and ultrasound segmentation—we demonstrate that fine-tuning consistently matches or exceeds the performance of training from scratch. Notably, fine-tuned models exhibit superior robustness when labeled data is limited and converge significantly faster, suggesting that transfer learning is a highly effective strategy for medical image analysis.

Introduction

Training deep Convolutional Neural Networks (CNNs) from scratch in the medical domain is often hindered by the scarcity of large-scale, expert-annotated datasets. While transfer learning from models trained on natural images (e.g., ImageNet) is common in computer vision, its applicability to medical imaging has been met with skepticism due to the significant domain gap between natural and clinical images. This paper systematically investigates whether fine-tuning pre-trained CNNs can eliminate the need for training from scratch across various medical modalities and tasks.

Methods

The researchers utilized the AlexNet architecture to compare three primary approaches:

  1. Handcrafted Features: Traditional computer vision methods tailored to specific clinical tasks.
  2. Training from Scratch: CNNs initialized with random weights and trained exclusively on medical datasets.
  3. Layer-wise Fine-tuning: Models pre-trained on ImageNet were incrementally updated. The study explored "shallow tuning" (updating only the final layers) versus "deep tuning" (updating all layers, including early convolutional ones).

Datasets and Applications

The methodology was validated across four distinct applications representing different modalities and task types:

  • Polyp Detection (Endoscopy): 40 colonoscopy videos (~19,000 frames).
  • Image Quality Assessment (Endoscopy): 4,000 frames labeled for informativeness.
  • Pulmonary Embolism (PE) Detection (CT): 121 CTPA datasets with 326 PEs.
  • Intima-media Boundary Segmentation (Ultrasound): 92 carotid artery videos.
Examples of polyp detection in colonoscopy frames, illustrating the visual complexity of the task.

Key Results

The experimental results consistently favored fine-tuning over training from scratch and handcrafted methods.

Performance and Tuning Depth

In all four applications, fine-tuned CNNs achieved performance levels equal to or better than scratch-trained models. For complex tasks like polyp detection, "deep tuning" (updating layers from conv1 through fc8) was essential for peak sensitivity. As shown in the FROC analysis below, the fine-tuned model (conv1-fc8) significantly outperformed the scratch model at low false-positive rates ($p < .05$).

Robustness to Data Scarcity

A pivotal finding was the robustness of fine-tuned models when training data was reduced. While models trained from scratch experienced "dramatic performance degradation" when using only 25% of the training data, fine-tuned models maintained high performance levels.

FROC analysis for polyp detection: (a) shows sensitivity improvements with deeper fine-tuning; (b) demonstrates the robustness of fine-tuning compared to training from scratch when data is limited.

Convergence and Efficiency

Fine-tuned models reached optimal performance significantly faster than those trained from scratch. This efficiency reduces both computational costs and the time required for model development.

Discussion and Conclusion

The study suggests that low-level features learned from natural images are "universal" enough to be highly effective for medical image analysis. Fine-tuning offers a practical solution to the "small data" problem in medicine, providing a more robust and efficient alternative to training from scratch.

Future work may explore modern architectures like ResNet or Vision Transformers, as well as the potential of unsupervised pre-training on large unlabeled medical datasets to further bridge the domain gap.

Summary updated

Abstract

Training a deep convolutional neural network (CNN) from scratch is difficult because it requires a large amount of labeled training data and a great deal of expertise to ensure proper convergence. A promising alternative is to fine-tune a CNN that has been pre-trained using, for instance, a large set of labeled natural images. However, the substantial differences between natural and medical images may advise against such knowledge transfer. In this paper, we seek to answer the following central question in the context of medical image analysis: \emph{Can the use of pre-trained deep CNNs with sufficient fine-tuning eliminate the need for training a deep CNN from scratch?} To address this question, we considered 4 distinct medical imaging applications in 3 specialties (radiology, cardiology, and gastroenterology) involving classification, detection, and segmentation from 3 different imaging modalities, and investigated how the performance of deep CNNs trained from scratch compared with the pre-trained CNNs fine-tuned in a layer-wise manner. Our experiments consistently demonstrated that (1) the use of a pre-trained CNN with adequate fine-tuning outperformed or, in the worst case, performed as well as a CNN trained from scratch; (2) fine-tuned CNNs were more robust to the size of training sets than CNNs trained from scratch; (3) neither shallow tuning nor deep tuning was the optimal choice for a particular application; and (4) our layer-wise fine-tuning scheme could offer a practical way to reach the best performance for the application at hand based on the amount of available data.
BibTeX
@article{tajbakhsh2016convolutional,
  author = {Nima Tajbakhsh and Jae Y. Shin and Suryakanth R. Gurudu and R. Todd Hurst and Christopher B. Kendall and Michael B. Gotway and Jianming Liang},
  title = {{Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning?}},
  journal = {IEEE Trans. Medical Imaging},
  year = {2016},
  url = {https://openreview.net/forum?id=qBv5HGBko4},
}
Figure