Abstract
Within medical imaging, manual curation of sufficient well-labeled samples is cost, time and scale-prohibitive. To improve the representativeness of the training dataset, for the first time, we present an approach to utilize large amounts of freely available web data through web-crawling. To handle noise and weak nature of web annotations, we propose a two-step transfer learning based training process with a robust loss function, termed as Webly Supervised Learning (WSL) to train deep models for the task. We also leverage search by image to improve the search specificity of our web-crawling and reduce cross-domain noise. Within WSL, we explicitly model the noise structure between classes and incorporate it to selectively distill knowledge from the web data during model training. To demonstrate improved performance due to WSL, we benchmarked on a publicly available 10-class fine-grained skin lesion classification dataset and report a significant improvement of top-1 classification accuracy from 71.25 % to 80.53 % due to the incorporation of web-supervision.
Introduction
Deep learning thrives on data, but in the medical world, data is expensive. Curating a high-quality, expert-labeled dataset for skin cancer (like melanoma) is time-consuming and doesn't scale easily. Small datasets often lead to "overfitting," where a model memorizes the training examples but fails to recognize new cases in the real world. This paper addresses the "data hunger" of deep networks by leveraging the internet as a source of training data through a framework called Webly Supervised Learning (WSL).
The Challenge of Web Data
While web data is plentiful, it is inherently "messy." The authors identify two main types of noise that can confuse a standard model:
- Cross-domain noise: Searching for medical terms often returns histology slides, medical diagrams, or unrelated icons rather than clinical photos.
- Cross-category noise: Images may be relevant to the domain but incorrectly labeled (e.g., a benign mole labeled as a tumor).
Methodology
The researchers proposed a two-step transfer learning process to effectively utilize this noisy data:
- Smart Crawling (Search by Image): Instead of keyword-only searches, they used "Search by Image" (via Google Images) with clean images as queries. This ensures the retrieved web images are visually similar to actual clinical photos, significantly reducing cross-domain noise.
- Noise Modeling: They developed a class-transition matrix ($T$) to mathematically model how likely one skin lesion class is to be confused with another on the web.
- Two-Step Training:
- Step A: An initial Inception V3 model was trained on the noisy web data using a robust loss function that incorporates the transition matrix $T$ to "diffuse" labels, acknowledging potential mislabeling.
- Step B: The "web-smart" model was then fine-tuned using the Dermofit Image Library, a clean, expert-labeled dataset of 1,300 images across 10 classes.
Results
The inclusion of web-supervision provided a significant performance boost:
- Top-1 Accuracy: Increased from 71.25% (clean data only) to 80.53% (WSL approach).
- Cohen’s Kappa: Improved from 0.650 to 0.768, indicating more reliable classification even with imbalanced data.
- AUC: The average AUC reached 0.970, showing excellent discriminative power.
Qualitatively, t-SNE visualizations show that the WSL-trained model creates much tighter, more distinct clusters for different diseases compared to the baseline, indicating superior feature learning.
Conclusion
This framework demonstrates that webly supervised learning can significantly enhance medical image classification, particularly for under-represented classes. By embracing and modeling noise rather than just attempting to clean it, models can become more resilient and accurate in data-scarce environments.
Summary updated