Abstract
Contrastive learning is a key technique of modern self-supervised learning. The broader accessibility of earlier approaches is hindered by the need for heavy computational resources (e.g., at least 8 GPUs or 32 TPU cores) to accommodate large-scale negative samples or momentum. The more recent SimSiam approach addresses such key limitations via stop-gradient without momentum encoders. In medical image analysis, multiple instances can be achieved from the same patient or tissue. Inspired by these advances, we propose a simple triplet representation learning (SimTriplet) approach for pathological images. The contribution of the paper is three-fold: (1) SimTriplet takes advantage of the multi-view nature of medical images beyond self-augmentation; (2) The method maximizes both intra-sample and inter-sample similarities via triplets from positive pairs, without using negative samples; and (3) Mixed precision training is employed to enable training on a single GPU with 16GB memory. By learning from 79,000 unlabeled pathological patch images, SimTriplet achieved 10.58% better performance compared with supervised learning and 2.13% better performance compared with SimSiam. Our proposed SimTriplet can achieve decent performance using only 1% labeled data. The code and data are available at https://github.com/hrlblab/SimTriple.
Introduction
Deep learning in digital pathology faces two major hurdles: the "GigaPixel" problem and the "Annotation" problem. Histopathology slides are massive, and obtaining expert labels is expensive and slow. Contrastive learning offers a solution by learning from unlabeled data, but popular methods like SimCLR or MoCo require massive computational clusters to handle large batches of negative samples.
While newer methods like SimSiam reduced these requirements, they do not fully exploit the unique spatial structure of medical data—specifically, the fact that patches of tissue located near each other are likely the same tissue type. We propose SimTriplet, a framework designed to train high-performance models on a single, consumer-grade GPU by leveraging these spatial relationships.
Methods
SimTriplet builds on the SimSiam architecture but introduces a spatial triplet input to capture both intra-sample and inter-sample similarities.
The Triplet Input
Instead of the standard approach of augmenting a single image twice, SimTriplet takes two adjacent patches ($m_1$ and $m_2$) from a whole slide image. It creates a triplet: two different augmentations of the first patch ($x_1, x_2$) and one augmentation of the neighbor ($x_3$).
Dual-Similarity Loss
The model is trained to maximize two types of similarity:
- Intra-sample similarity: Ensuring different augmentations of the same patch ($x_1, x_2$) result in similar feature representations.
- Inter-sample similarity: Ensuring a patch and its spatial neighbor ($x_2, x_3$) are mapped closely in the feature space.
Implementation and Efficiency
To enable training on a single 16GB GPU, we utilize Mixed Precision Training, performing calculations in 16-bit precision to save memory. The architecture uses a ResNet-50 backbone with a 3-layer MLP projection head and a predictor. A stop-gradient operation is applied to two of the three paths to prevent model collapse without requiring momentum encoders or negative samples.
Experimental Setup
The study utilized data from The Cancer Genome Atlas (TCGA), focusing on melanoma:
- Unlabeled Data: 79,000 pathological patch images from 79 Whole Slide Images (WSIs).
- Annotated Data: 4,698 images from 5 WSIs, labeled with eight tissue types (e.g., tumor, stroma, epidermis).
- Test Set: 1,921 images from 2 separate WSIs to evaluate generalization.
Results
SimTriplet demonstrates significant improvements in both accuracy and data efficiency.
- Superior Accuracy: SimTriplet achieved a 71.71% Balanced Accuracy, outperforming standard supervised learning (61.13%) and SimSiam (69.58%).
- Low-Data Regime: When only 1% of the labeled data was used, SimTriplet maintained a 70.90% accuracy. In contrast, supervised learning plummeted to 6.14%, demonstrating the power of self-supervised pre-training.
- Resource Efficiency: Unlike MoCo or SimCLR which require 8-64 GPUs, SimTriplet achieves state-of-the-art results using only a single GPU.
Conclusion
SimTriplet provides an efficient and effective way to leverage the spatial context inherent in pathology slides as "free" supervision. By avoiding negative samples and momentum encoders, it democratizes high-end medical AI research. While currently focused on 2D tissue patches, future work could extend this to 3D volumetric data or incorporate more advanced sampling to handle tissue boundaries.
Summary updated