Self-Supervised Contrastive Video-Speech Representation Learning for Ultrasound

Jianbo Jiao, Yifan Cai, Mohammad Alsharid, Lior Drukker, Aris T. Papageorghiou, J. Alison Noble

Introduction

Deep learning has significantly advanced medical imaging, yet it remains constrained by a heavy reliance on large-scale, manually annotated datasets. In specialized fields like fetal ultrasound, obtaining these labels is particularly challenging, requiring expert clinicians to identify complex anatomical structures within noisy, abstract acoustic data.

To address this "data hunger," researchers from the University of Oxford have introduced a self-supervised framework that leverages a naturally occurring, "free" source of information: the sonographer’s narrative speech recorded during routine examinations. By learning the inherent correlations between ultrasound video and spoken descriptions, the model develops robust anatomical representations without the need for explicit manual labeling.

Conceptual overview of cross-modal correlation between ultrasound video and sonographer speech in a shared embedding space.

Methodology

The core of the proposed approach is a multi-modal learning framework designed to align visual and auditory information.

Cross-modal Contrastive Learning

The model employs a contrastive loss to map video clips and their corresponding audio segments into a shared embedding space. It is trained to maximize the similarity between "positive" pairs (video and speech from the same temporal window) while minimizing similarity for "negative" pairs.

Hard-Pair Sampling and Architecture

To enhance the model's discriminative power, the researchers implemented a hard-pair sampling strategy. This involves selecting "hard-negative" and "hard-positive" pairs from proximal timestamps, forcing the network to distinguish between subtle temporal and anatomical variations. The architecture utilizes a ResNeXt-50 backbone with Squeeze-and-Excitation modules to process both video frames and 2D log-spectrograms of the audio.

The framework architecture and sampling strategy, illustrating how positive, hard-positive, and negative pairs are selected across the video-speech timeline.

Affinity-Aware Self-paced Learning

Clinical audio is often cluttered with non-narrative noise. The framework incorporates an energy-based voice activity detection (VAD) algorithm to establish a learning curriculum. "High-affinity" segments containing clear speech are prioritized for cross-modal contrastive learning. Conversely, "low-affinity" segments (noise or silence) are utilized for a secondary, video-only pretext task—predicting the correct order of shuffled frames—ensuring the model learns from the entire video stream.

Experimental Results

The model was evaluated on two critical downstream tasks: Standard Plane Detection (identifying 14 anatomical views) and Eye-Gaze Saliency Prediction (predicting sonographer visual attention).

  • Plane Detection: The self-supervised model achieved an F1-score of 72.6%, surpassing video-only self-supervision (71.5%) and performing comparably to models pre-trained on ImageNet (72.5%). This highlights the advantage of domain-specific pre-training for medical data.
  • Eye-Gaze Prediction: Remarkably, the self-supervised approach outperformed the supervised SonoNet model across several metrics, including KL divergence (3.05 vs. 3.14) and Correlation Coefficient (0.22 vs. 0.21).

Discussion and Conclusion

This research demonstrates that sonographer speech serves as a powerful, sparse label for ultrasound video. By managing the temporal "jitter" and noise inherent in clinical environments through a self-paced curriculum, the framework learns representations that generalize well to complex clinical tasks. While the effectiveness of the model depends on the presence of descriptive narration, this approach paves the way for utilizing vast amounts of unlabelled clinical data to improve diagnostic AI tools. Future work may extend this multi-modal paradigm to other medical imaging modalities where expert narration is available.

Summary updated

BibTeX
@inproceedings{jiao2020self,
  author = {Jianbo Jiao and Yifan Cai and Mohammad Alsharid and Lior Drukker and Aris T. Papageorghiou and J. Alison Noble},
  title = {{Self-Supervised Contrastive Video-Speech Representation Learning for Ultrasound}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2020},
  doi = {10.1007/978-3-030-59716-0\_51},
}
Figure