SimTriplet: Simple Triplet Representation Learning with a Single GPU

Quan Liu, Peter C. Louis, Yuzhe Lu, Aadarsh Jha, Mengyang Zhao, Ruining Deng, Tianyuan Yao, Joseph T. Roland, Haichun Yang, Shilin Zhao, Lee E. Wheless, Yuankai Huo

TL;DR

SimTriplet is a self-supervised learning framework designed to train high-performance medical image models on a single, consumer-grade GPU. By leveraging the spatial relationships between adjacent tissue patches in histopathology slides, it achieves superior accuracy over supervised learning and existing contrastive methods while using only 1% of labeled data.

What Problem Does This Paper Address?

Deep learning in digital pathology faces two major hurdles: the "GigaPixel" problem and the "Annotation" problem. Histopathology slides are massive, and getting expert pathologists to provide pixel-wise labels is expensive and slow.

To bypass this, researchers use contrastive learning, where a model learns by comparing images without labels. However, popular methods like SimCLR or MoCo require massive computational clusters (8+ GPUs) to handle the large "negative samples" needed to prevent the model from taking shortcuts. While newer methods like SimSiam reduced these requirements, they don't fully exploit the unique structure of medical data—specifically, the fact that patches of tissue located near each other are almost certainly the same tissue type.

Methods

The authors introduce SimTriplet, which builds on the SimSiam architecture but adds a spatial twist.

  1. The Triplet Input: Instead of taking one image and augmenting it twice (the standard approach), SimTriplet takes two adjacent patches ($m_1$ and $m_2$) from a whole slide image. It creates a triplet: two different augmentations of the first patch and one augmentation of the neighbor.
  2. Dual-Similarity Loss: The model is trained to maximize two types of similarity:
  3. Intra-sample: Ensuring the two versions of the same patch look similar in the feature space.
  4. Inter-sample: Ensuring the patch and its neighbor look similar.
  5. Efficiency Hacks: To fit this on a single 16GB GPU, the authors used Mixed Precision Training, which performs calculations in half-precision (16-bit) to save memory without sacrificing model accuracy.
  6. Architecture: They use a standard ResNet-50 backbone with a 3-layer MLP projection head and a predictor. Crucially, they use a stop-gradient operation on two of the three paths to prevent "model collapse" (where the model outputs a constant value for everything).

Datasets

The study utilized data from The Cancer Genome Atlas (TCGA), specifically focusing on melanoma skin cancer:

  • Unlabeled Data: 79,000 pathological patch images extracted from 79 Whole Slide Images (WSIs).
  • Annotated Data: A smaller set of 4,698 images from 5 WSIs, labeled with eight tissue types (e.g., tumor, stroma, blood vessel, epidermis).
  • Test Set: 1,921 images from 2 separate WSIs to ensure the model generalizes to new patients.

Key Insights and Ideas

  • Spatial Context is "Free" Supervision: In pathology, the "multi-view" nature of the data isn't just about rotating or cropping an image; it's about the physical proximity of tissue. SimTriplet treats nearby patches as positive pairs, providing a much stronger training signal than self-augmentation alone.
  • No Negatives Needed: By adopting the SimSiam philosophy, SimTriplet avoids the need for "negative samples" (comparing an image to thousands of different images). This is what allows the model to run on a single GPU rather than a server farm.
  • Robustness to Low Data: The most striking insight is how well the model performs when labels are scarce. Because the model learns the "language" of tissue types through 79,000 unlabeled patches, it only needs a tiny handful of labels to map those features to specific categories.

Key Results

  • Superior Accuracy: SimTriplet achieved a 71.71% Balanced Accuracy, outperforming standard supervised learning (61.13%) and the previous state-of-the-art SimSiam (69.58%).
  • The 1% Miracle: When only 1% of the labeled data was used, SimTriplet maintained a 70.90% accuracy. In contrast, supervised learning completely failed in this low-data regime, plummeting to just 6.14%.
  • Efficiency: The entire system was trained on a single GPU with 16GB of memory, making high-end medical AI research accessible to smaller labs and individual researchers.

Limitations and Future Work

While highly effective for pathology, the paper focuses specifically on 2D tissue patches. The authors do not discuss how this might translate to 3D medical imaging (like CT or MRI scans) where "adjacency" is volumetric. Additionally, the assumption that adjacent patches are the same tissue type is generally true but can fail at the exact boundaries between different tissues (e.g., the edge of a tumor). Future work could involve more sophisticated sampling strategies to handle these boundary conditions.

Summary updated

Abstract

Contrastive learning is a key technique of modern self-supervised learning. The broader accessibility of earlier approaches is hindered by the need of heavy computational resources (e.g., at least 8 GPUs or 32 TPU cores), which accommodate for large-scale negative samples or momentum. The more recent SimSiam approach addresses such key limitations via stop-gradient without momentum encoders. In medical image analysis, multiple instances can be achieved from the same patient or tissue. Inspired by these advances, we propose a simple triplet representation learning (SimTriplet) approach on pathological images. The contribution of the paper is three-fold: (1) The proposed SimTriplet method takes advantage of the multi-view nature of medical images beyond self-augmentation; (2) The method maximizes both intra-sample and inter-sample similarities via triplets from positive pairs, without using negative samples; and (3) The recent mix precision training is employed to advance the training by only using a single GPU with 16 GB memory. By learning from 79,000 unlabeled pathological patch images, SimTriplet achieved 10.58% better performance compared with supervised learning. It also achieved 2.13% better performance compared with SimSiam. Our proposed SimTriplet can achieve decent performance using only 1% labeled data. The code and data are available at https://github.com/hrlblab/SimTriplet .
BibTeX
@inproceedings{liu2021simtriplet,
  author = {Quan Liu and Peter C. Louis and Yuzhe Lu and Aadarsh Jha and Mengyang Zhao and Ruining Deng and Tianyuan Yao and Joseph T. Roland and Haichun Yang and Shilin Zhao and Lee E. Wheless and Yuankai Huo},
  title = {{SimTriplet: Simple Triplet Representation Learning with a Single GPU}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2021},
  url = {https://openreview.net/forum?id=gziqZSSoK_7},
}

Loading related content...

Figure