Contrastive Self-Supervised Learning for Spatio-Temporal Analysis of Lung Ultrasound Videos

Li Chen, Jonathan Rubin, Jiahong Ouyang, Naveen Balaraju, Shubham Patil, Courosh Mehanian, Sourabh Kulhare, Rachel Millin, Kenton W. Gregory, Cynthia Gregory, Meihua Zhu, David O. Kessler, Laurie Malia, Almaz Dessie, Joni Rabiner, Di Coneybeare, Bo Shopsin, Andrew Hersh, Cristian Madar, Jeffrey Shupp, Laura S. Johnson, Jacob Avila, Kristin Dwyer, Peter Weimersheimer, Balasundar Raju, Jochen Kruecker, Alvin ChenISBI 2023 · DOI

TL;DR

This paper introduces a novel contrastive self-supervised learning (SSL) approach for analyzing lung ultrasound videos, enabling the extraction of meaningful spatio-temporal features without requiring extensive manual annotations. Their method significantly improves downstream tasks like lung consolidation detection, especially in low-data regimes, demonstrating the power of SSL for medical imaging.

What Problem Does This Paper Address?

Medical imaging, particularly ultrasound, often requires expert annotations for training machine learning models. However, annotating ultrasound videos is incredibly time-consuming and expensive, creating a bottleneck for developing robust diagnostic tools. This paper addresses the challenge of building effective models for lung ultrasound analysis when labeled data is scarce, aiming to leverage the vast amounts of unlabeled ultrasound video data available.

Methods

The core of this work is an adaptation of state-of-the-art contrastive SSL methods, specifically BYOL, for 2D+time medical ultrasound videos. The key innovations include:

  • Modified 3D Encoder: A 3D convolutional neural network (based on Darknet-53) is used to capture spatio-temporal information from video sequences, moving beyond the 2D focus of many existing SSL methods.
  • Domain-Specific Spatio-Temporal Augmentations: Novel augmentation techniques are introduced that mimic the variability inherent in ultrasound imaging (e.g., random affine transformations, color jitter, Gaussian noise, random erasing, and temporal shuffling). These augmentations are crucial for generating the positive pairs needed for contrastive learning without relying on external labels.
  • Two-Stream Architecture: Similar to BYOL, the method employs an "online" network and a "target" network. The online network learns representations from augmented video clips, while the target network's weights are an exponential moving average (EMA) of the online network's weights, providing a stable learning target.
  • Downstream Tasks: The learned representations are evaluated on two key tasks:
  • Saliency Map Generation: The encoder is used to highlight regions of interest, which are then assessed for their ability to localize lung consolidations.
  • Video Classification: A fully-connected layer is appended to the encoder for classifying lung consolidations, with experiments comparing "SSL Feature Extractor" (encoder weights fixed) and "SSL Fine-Tuned" (encoder weights updated) approaches.

Datasets

The study utilizes a large, multi-center dataset comprising 27,063 lung ultrasound videos from 528 patients across 8 clinical sites.

  • Unlabeled Data: 25,394 videos were used for SSL pre-training.
  • Labeled Data: 1,669 videos were annotated for the presence or absence of lung consolidation. This labeled set was further split into training, validation, and test sets, with the training set being progressively reduced to evaluate performance in low-data regimes.

Key Insights and Ideas

  • SSL for Spatio-Temporal Medical Data: The paper successfully demonstrates that contrastive SSL, previously dominant in 2D natural image analysis, can be effectively extended to 3D spatio-temporal medical video data.
  • Augmentations are Key: The introduction of domain-specific spatio-temporal augmentations is critical for making contrastive learning work on ultrasound videos, capturing the unique challenges and variability of this modality.
  • Robustness in Low-Data Regimes: The most striking finding is the remarkable resilience of SSL-pretrained models when the amount of labeled training data is severely limited (as low as 5%). This highlights SSL's potential to democratize AI development in medical imaging.
  • Improved Localization: Saliency maps generated by the SSL models are more specific and better localize pathology compared to baseline models, suggesting richer feature extraction.

Key Results

  • Saliency Map Generation: SSL-based models ("SSL Feature Extractor" and "SSL Fine-Tuned") produced saliency maps with significantly higher weighted IOU (0.16 and 0.24 respectively) compared to "Random Feature Extractor" (0.01) and "Fully-Supervised" (0.01) models, indicating better pathology localization.
  • Performance with Limited Data: When labeled training data was reduced to 5%, the "Fully-Supervised" model's accuracy dropped dramatically, while the SSL-based models maintained consistent performance. For example, at 5% labeled data, SSL models achieved an accuracy of around 0.82-0.84, whereas the fully-supervised model dropped to around 0.6.
  • Comparison to Baselines: SSL-pretrained models consistently outperformed baseline models trained without SSL pretraining, especially in low-data scenarios.
  • Alternative SSL Algorithms: The proposed spatio-temporal augmentation strategy also showed positive results when applied with an alternative contrastive SSL algorithm (SimCLR), suggesting the generality of the augmentation approach.

Limitations and Future Work

  • Temporal Augmentation Impact: The paper notes that SSL training without temporal augmentations led to a 0.02 decrease in AUC, suggesting that while spatio-temporal augmentations are beneficial, their specific design might be further optimized.
  • Generalizability to Other Modalities: While successful for lung ultrasound, the generalizability of these specific spatio-temporal augmentations to other medical video modalities (e.g., cardiac ultrasound, endoscopy) is not explored.
  • Computational Cost: 3D convolutions and extensive augmentations can be computationally intensive, which might be a consideration for real-time applications or deployment on resource-constrained devices.

Future work could involve exploring more advanced spatio-temporal augmentation strategies, investigating the impact of different SSL frameworks, and evaluating the method on a wider range of medical video tasks and modalities.

Summary updated

BibTeX
@inproceedings{chen2023contrastive,
  author = {Li Chen and Jonathan Rubin and Jiahong Ouyang and Naveen Balaraju and Shubham Patil and Courosh Mehanian and Sourabh Kulhare and Rachel Millin and Kenton W. Gregory and Cynthia Gregory and Meihua Zhu and David O. Kessler and Laurie Malia and Almaz Dessie and Joni Rabiner and Di Coneybeare and Bo Shopsin and Andrew Hersh and Cristian Madar and Jeffrey Shupp and Laura S. Johnson and Jacob Avila and Kristin Dwyer and Peter Weimersheimer and Balasundar Raju and Jochen Kruecker and Alvin Chen},
  title = {{Contrastive Self-Supervised Learning for Spatio-Temporal Analysis of Lung Ultrasound Videos}},
  booktitle = {ISBI},
  year = {2023},
  doi = {10.1109/isbi53787.2023.10230816},
}
Figure