Abstract
This study addresses a critical question in medical imaging: whether deep learning models should be trained from scratch on domain-specific data or fine-tuned from pre-trained models on natural images. By evaluating four diverse clinical tasks—polyp detection, image quality assessment, pulmonary embolism detection, and ultrasound segmentation—we demonstrate that fine-tuning consistently matches or exceeds the performance of training from scratch. Notably, fine-tuned models exhibit superior robustness when labeled data is limited and converge significantly faster, suggesting that transfer learning is a highly effective strategy for medical image analysis.
Introduction
Training deep Convolutional Neural Networks (CNNs) from scratch in the medical domain is often hindered by the scarcity of large-scale, expert-annotated datasets. While transfer learning from models trained on natural images (e.g., ImageNet) is common in computer vision, its applicability to medical imaging has been met with skepticism due to the significant domain gap between natural and clinical images. This paper systematically investigates whether fine-tuning pre-trained CNNs can eliminate the need for training from scratch across various medical modalities and tasks.
Methods
The researchers utilized the AlexNet architecture to compare three primary approaches:
- Handcrafted Features: Traditional computer vision methods tailored to specific clinical tasks.
- Training from Scratch: CNNs initialized with random weights and trained exclusively on medical datasets.
- Layer-wise Fine-tuning: Models pre-trained on ImageNet were incrementally updated. The study explored "shallow tuning" (updating only the final layers) versus "deep tuning" (updating all layers, including early convolutional ones).
Datasets and Applications
The methodology was validated across four distinct applications representing different modalities and task types:
- Polyp Detection (Endoscopy): 40 colonoscopy videos (~19,000 frames).
- Image Quality Assessment (Endoscopy): 4,000 frames labeled for informativeness.
- Pulmonary Embolism (PE) Detection (CT): 121 CTPA datasets with 326 PEs.
- Intima-media Boundary Segmentation (Ultrasound): 92 carotid artery videos.
Key Results
The experimental results consistently favored fine-tuning over training from scratch and handcrafted methods.
Performance and Tuning Depth
In all four applications, fine-tuned CNNs achieved performance levels equal to or better than scratch-trained models. For complex tasks like polyp detection, "deep tuning" (updating layers from conv1 through fc8) was essential for peak sensitivity. As shown in the FROC analysis below, the fine-tuned model (conv1-fc8) significantly outperformed the scratch model at low false-positive rates ($p < .05$).
Robustness to Data Scarcity
A pivotal finding was the robustness of fine-tuned models when training data was reduced. While models trained from scratch experienced "dramatic performance degradation" when using only 25% of the training data, fine-tuned models maintained high performance levels.
Convergence and Efficiency
Fine-tuned models reached optimal performance significantly faster than those trained from scratch. This efficiency reduces both computational costs and the time required for model development.
Discussion and Conclusion
The study suggests that low-level features learned from natural images are "universal" enough to be highly effective for medical image analysis. Fine-tuning offers a practical solution to the "small data" problem in medicine, providing a more robust and efficient alternative to training from scratch.
Future work may explore modern architectures like ResNet or Vision Transformers, as well as the potential of unsupervised pre-training on large unlabeled medical datasets to further bridge the domain gap.
Summary updated