SimTriplet: Simple Triplet Representation Learning with a Single GPU

Quan Liu, Peter C. Louis, Yuzhe Lu, Aadarsh Jha, Mengyang Zhao, Ruining Deng, Tianyuan Yao, Joseph T. Roland, Haichun Yang, Shilin Zhao, Lee E. Wheless, Yuankai Huo

Abstract

Contrastive learning is a key technique of modern self-supervised learning. The broader accessibility of earlier approaches is hindered by the need for heavy computational resources (e.g., at least 8 GPUs or 32 TPU cores) to accommodate large-scale negative samples or momentum. The more recent SimSiam approach addresses such key limitations via stop-gradient without momentum encoders. In medical image analysis, multiple instances can be achieved from the same patient or tissue. Inspired by these advances, we propose a simple triplet representation learning (SimTriplet) approach for pathological images. The contribution of the paper is three-fold: (1) SimTriplet takes advantage of the multi-view nature of medical images beyond self-augmentation; (2) The method maximizes both intra-sample and inter-sample similarities via triplets from positive pairs, without using negative samples; and (3) Mixed precision training is employed to enable training on a single GPU with 16GB memory. By learning from 79,000 unlabeled pathological patch images, SimTriplet achieved 10.58% better performance compared with supervised learning and 2.13% better performance compared with SimSiam. Our proposed SimTriplet can achieve decent performance using only 1% labeled data. The code and data are available at https://github.com/hrlblab/SimTriple.

Introduction

Deep learning in digital pathology faces two major hurdles: the "GigaPixel" problem and the "Annotation" problem. Histopathology slides are massive, and obtaining expert labels is expensive and slow. Contrastive learning offers a solution by learning from unlabeled data, but popular methods like SimCLR or MoCo require massive computational clusters to handle large batches of negative samples.

Comparison of SimTriplet with other self-supervised learning frameworks and their resource requirements.

While newer methods like SimSiam reduced these requirements, they do not fully exploit the unique spatial structure of medical data—specifically, the fact that patches of tissue located near each other are likely the same tissue type. We propose SimTriplet, a framework designed to train high-performance models on a single, consumer-grade GPU by leveraging these spatial relationships.

Methods

SimTriplet builds on the SimSiam architecture but introduces a spatial triplet input to capture both intra-sample and inter-sample similarities.

Overview of the SimTriplet framework, including patch extraction from whole slide images and the triplet architecture.

The Triplet Input

Instead of the standard approach of augmenting a single image twice, SimTriplet takes two adjacent patches ($m_1$ and $m_2$) from a whole slide image. It creates a triplet: two different augmentations of the first patch ($x_1, x_2$) and one augmentation of the neighbor ($x_3$).

Dual-Similarity Loss

The model is trained to maximize two types of similarity:

  • Intra-sample similarity: Ensuring different augmentations of the same patch ($x_1, x_2$) result in similar feature representations.
  • Inter-sample similarity: Ensuring a patch and its spatial neighbor ($x_2, x_3$) are mapped closely in the feature space.

Implementation and Efficiency

To enable training on a single 16GB GPU, we utilize Mixed Precision Training, performing calculations in 16-bit precision to save memory. The architecture uses a ResNet-50 backbone with a 3-layer MLP projection head and a predictor. A stop-gradient operation is applied to two of the three paths to prevent model collapse without requiring momentum encoders or negative samples.

Experimental Setup

The study utilized data from The Cancer Genome Atlas (TCGA), focusing on melanoma:

  • Unlabeled Data: 79,000 pathological patch images from 79 Whole Slide Images (WSIs).
  • Annotated Data: 4,698 images from 5 WSIs, labeled with eight tissue types (e.g., tumor, stroma, epidermis).
  • Test Set: 1,921 images from 2 separate WSIs to evaluate generalization.

Results

SimTriplet demonstrates significant improvements in both accuracy and data efficiency.

  • Superior Accuracy: SimTriplet achieved a 71.71% Balanced Accuracy, outperforming standard supervised learning (61.13%) and SimSiam (69.58%).
  • Low-Data Regime: When only 1% of the labeled data was used, SimTriplet maintained a 70.90% accuracy. In contrast, supervised learning plummeted to 6.14%, demonstrating the power of self-supervised pre-training.
  • Resource Efficiency: Unlike MoCo or SimCLR which require 8-64 GPUs, SimTriplet achieves state-of-the-art results using only a single GPU.

Conclusion

SimTriplet provides an efficient and effective way to leverage the spatial context inherent in pathology slides as "free" supervision. By avoiding negative samples and momentum encoders, it democratizes high-end medical AI research. While currently focused on 2D tissue patches, future work could extend this to 3D volumetric data or incorporate more advanced sampling to handle tissue boundaries.

Summary updated

Abstract

Contrastive learning is a key technique of modern self-supervised learning. The broader accessibility of earlier approaches is hindered by the need of heavy computational resources (e.g., at least 8 GPUs or 32 TPU cores), which accommodate for large-scale negative samples or momentum. The more recent SimSiam approach addresses such key limitations via stop-gradient without momentum encoders. In medical image analysis, multiple instances can be achieved from the same patient or tissue. Inspired by these advances, we propose a simple triplet representation learning (SimTriplet) approach on pathological images. The contribution of the paper is three-fold: (1) The proposed SimTriplet method takes advantage of the multi-view nature of medical images beyond self-augmentation; (2) The method maximizes both intra-sample and inter-sample similarities via triplets from positive pairs, without using negative samples; and (3) The recent mix precision training is employed to advance the training by only using a single GPU with 16GB memory. By learning from 79,000 unlabeled pathological patch images, SimTriplet achieved 10.58% better performance compared with supervised learning. It also achieved 2.13% better performance compared with SimSiam. Our proposed SimTriplet can achieve decent performance using only 1% labeled data. The code and data are available at https://github.com/hrlblab/SimTriple.
BibTeX
@inproceedings{liu2021simtriplet,
  author = {Quan Liu and Peter C. Louis and Yuzhe Lu and Aadarsh Jha and Mengyang Zhao and Ruining Deng and Tianyuan Yao and Joseph T. Roland and Haichun Yang and Shilin Zhao and Lee E. Wheless and Yuankai Huo},
  title = {{SimTriplet: Simple Triplet Representation Learning with a Single GPU}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2021},
  url = {https://openreview.net/forum?id=gziqZSSoK_7},
}
Figure