Towards Scalable Foundation Models for Digital Dermatology

Fabian Gröger, Philippe Gottfrois, Ludovic Amruthalingam, Álvaro González-Jiménez, Simone Lionetti, Luis R. Soenksen-Martinez, Alexander A. Navarini, Marc PoulyarXiv 2024 · DOI · arXiv

Abstract

Digital dermatology is often constrained by the high cost of expert-labeled data and the computational demands of large-scale foundation models. This study introduces domain-specific foundation models developed using self-supervised learning on over 240,000 unlabeled skin images. By leveraging architectures like ViT-Tiny, these models achieve performance comparable to state-of-the-art models 50 times their size, such as MONET. Crucially, these "tiny" models effectively double the utility of labeled data, demonstrating superior performance in low-data regimes and significantly faster inference speeds, making them highly suitable for real-world clinical deployment.

The Challenge in Digital Dermatology

Digital dermatology faces a significant bottleneck: high-quality, expert-labeled data is expensive and difficult to obtain. While transfer learning from models trained on natural images (like ImageNet) is the industry standard, a substantial domain gap exists between general-purpose imagery and specialized dermoscopic images. Furthermore, existing high-performance medical foundation models, such as MONET, are often massive, requiring computational resources that are frequently unavailable in clinical or teledermatology settings. This research investigates whether smaller, domain-specific models can bridge this gap, providing high diagnostic accuracy while remaining scalable for real-world deployment.

Methodology and Data

The researchers employed Self-Supervised Learning (SSL) to learn meaningful representations from raw data without human-provided labels. They focused on smaller backbones, specifically ResNet-50 and ViT-Tiny, testing several state-of-the-art SSL strategies including SimCLR, BYOL, DINO, and iBOT.

Models were pre-trained on a curated collection of 242,039 unlabeled images from both public (e.g., ISIC, MED-NODE) and private sources. Evaluation was conducted across 12 diagnostic tasks from 9 datasets, including diverse skin tones (DDI) and pigment-rich skin from Sub-Saharan Africa (PASSION).

Key Results and Insights

Domain Specificity vs. Model Size

A primary finding is that domain specificity can outweigh model size. The "tiny" DINO model (5M parameters) trained on skin images often matched or exceeded the performance of the giant MONET model (304M parameters) across various tasks. This suggests that for specialized medical fields, the relevance of the training data is more critical than the sheer number of parameters.

Performance comparison across 12 diagnostic tasks. The domain-specific DINO (ViT-T/16) model matches or exceeds the much larger MONET (ViT-L/14) and the standard ImageNet baseline.

The "Label Multiplier" and Low-Data Efficiency

The authors identified a "utility score," finding that their DINO model reduces the need for annotated samples by a factor of 2 compared to ImageNet-pretrained models. In scenarios with very few labels (e.g., 1–10 samples per class), the domain-specific DINO model showed much steeper performance gains than domain-agnostic baselines.

Macro F1 Score across varying numbers of labeled samples per class. Domain-specific pre-training significantly improves performance in low-data regimes, effectively doubling the utility of labeled data.

Efficiency and Deployment

Beyond accuracy, the smaller models offer massive advantages in inference speed. The ViT-Tiny models achieved inference times of approximately 277 images/second, whereas MONET required roughly 3796 ms/image. This difference is critical for real-time clinical applications and deployment on edge devices.

Conclusion and Future Work

While these models generalize well to diverse datasets, performance remains a challenge on non-standardized images of pigment-rich skin (e.g., PAS-I and PAS-C). Future work will focus on including more diverse populations in pre-training sets to ensure equity in AI-driven dermatology and exploring full fine-tuning for specific clinical applications. The authors have publicly released their training code and model checkpoints to support the research community.

Summary updated

Abstract

The growing demand for accurate and equitable AI models in digital dermatology faces a significant challenge: the lack of diverse, high-quality labeled data. In this work, we investigate the potential of domain-specific foundation models for dermatology in addressing this challenge. We utilize self-supervised learning (SSL) techniques to pre-train models on a dataset of over 240,000 dermatological images from public and private collections. Our study considers several SSL methods and compares the resulting foundation models against domain-agnostic models like those pre-trained on ImageNet and state-of-the-art models such as MONET across 12 downstream tasks. Unlike previous research, we emphasize the development of smaller models that are more suitable for resource-limited clinical settings, facilitating easier adaptation to a broad range of use cases. Results show that models pre-trained in this work not only outperform general-purpose models but also approach the performance of models 50 times larger on clinically relevant diagnostic tasks. To promote further research in this direction, we publicly release both the training code and the foundation models, which can benefit clinicians in dermatological applications.
BibTeX
@misc{groger2024towards,
  author = {Fabian Gröger and Philippe Gottfrois and Ludovic Amruthalingam and Álvaro González-Jiménez and Simone Lionetti and Luis R. Soenksen-Martinez and Alexander A. Navarini and Marc Pouly},
  title = {{Towards Scalable Foundation Models for Digital Dermatology}},
  year = {2024},
  eprint = {2411.05514},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.05514},
}
Figure