Abstract
Digital dermatology is often constrained by the high cost of expert-labeled data and the computational demands of large-scale foundation models. This study introduces domain-specific foundation models developed using self-supervised learning on over 240,000 unlabeled skin images. By leveraging architectures like ViT-Tiny, these models achieve performance comparable to state-of-the-art models 50 times their size, such as MONET. Crucially, these "tiny" models effectively double the utility of labeled data, demonstrating superior performance in low-data regimes and significantly faster inference speeds, making them highly suitable for real-world clinical deployment.
The Challenge in Digital Dermatology
Digital dermatology faces a significant bottleneck: high-quality, expert-labeled data is expensive and difficult to obtain. While transfer learning from models trained on natural images (like ImageNet) is the industry standard, a substantial domain gap exists between general-purpose imagery and specialized dermoscopic images. Furthermore, existing high-performance medical foundation models, such as MONET, are often massive, requiring computational resources that are frequently unavailable in clinical or teledermatology settings. This research investigates whether smaller, domain-specific models can bridge this gap, providing high diagnostic accuracy while remaining scalable for real-world deployment.
Methodology and Data
The researchers employed Self-Supervised Learning (SSL) to learn meaningful representations from raw data without human-provided labels. They focused on smaller backbones, specifically ResNet-50 and ViT-Tiny, testing several state-of-the-art SSL strategies including SimCLR, BYOL, DINO, and iBOT.
Models were pre-trained on a curated collection of 242,039 unlabeled images from both public (e.g., ISIC, MED-NODE) and private sources. Evaluation was conducted across 12 diagnostic tasks from 9 datasets, including diverse skin tones (DDI) and pigment-rich skin from Sub-Saharan Africa (PASSION).
Key Results and Insights
Domain Specificity vs. Model Size
A primary finding is that domain specificity can outweigh model size. The "tiny" DINO model (5M parameters) trained on skin images often matched or exceeded the performance of the giant MONET model (304M parameters) across various tasks. This suggests that for specialized medical fields, the relevance of the training data is more critical than the sheer number of parameters.
The "Label Multiplier" and Low-Data Efficiency
The authors identified a "utility score," finding that their DINO model reduces the need for annotated samples by a factor of 2 compared to ImageNet-pretrained models. In scenarios with very few labels (e.g., 1–10 samples per class), the domain-specific DINO model showed much steeper performance gains than domain-agnostic baselines.
Efficiency and Deployment
Beyond accuracy, the smaller models offer massive advantages in inference speed. The ViT-Tiny models achieved inference times of approximately 277 images/second, whereas MONET required roughly 3796 ms/image. This difference is critical for real-time clinical applications and deployment on edge devices.
Conclusion and Future Work
While these models generalize well to diverse datasets, performance remains a challenge on non-standardized images of pigment-rich skin (e.g., PAS-I and PAS-C). Future work will focus on including more diverse populations in pre-training sets to ensure equity in AI-driven dermatology and exploring full fine-tuning for specific clinical applications. The authors have publicly released their training code and model checkpoints to support the research community.
Summary updated