Distilling Missing Modality Knowledge from Ultrasound for Endometriosis Diagnosis with Magnetic Resonance Images

Yuan Zhang, Hu Wang, David Butler, Minh-Son To, Jodie Avery, M. Louise Hull, Gustavo Carneiro

TL;DR

This paper tackles the challenge of diagnosing endometriosis from MRI, which is harder than from ultrasound (TVUS). They propose a novel knowledge distillation method to transfer diagnostic knowledge from a well-performing TVUS model to an MRI model, significantly improving MRI-based diagnosis accuracy even with unpaired data.

What Problem Does This Paper Address?

Endometriosis, a chronic gynecological disorder, can cause significant pain and infertility. A key diagnostic indicator is the obliteration of the pouch of Douglas (POD). While both Transvaginal Ultrasound (TVUS) and Magnetic Resonance Imaging (MRI) can detect this, TVUS is generally more accurate and easier to interpret for POD obliteration. However, patients often don't get both scans, and relying solely on MRI for POD obliteration detection is less accurate. This paper addresses the gap in accurate, single-modality MRI diagnosis by leveraging the superior diagnostic power of TVUS, even when the TVUS and MRI datasets are not directly paired (i.e., not from the same patient at the same time).

Methods

The core of the proposed method is a two-stage knowledge distillation process:

  1. Pre-training:
    • Teacher Model (TVUS): A ResNet(2+1)D model is pre-trained on TVUS video clips to detect POD obliteration. This model acts as the "teacher."
    • Student Model (MRI): A 3D Masked Auto-Encoder (MAE) based on a 3D Vision Transformer (3D ViT) is pre-trained on a large dataset of unlabeled 3D pelvic MRI volumes. This self-supervised pre-training helps the MRI encoder learn robust representations. After pre-training, a linear layer and a classification layer are added to form the initial student model.
  2. Knowledge Distillation:
    • The pre-trained TVUS teacher model is frozen.
  3. The pre-trained MRI student model is then fine-tuned using a knowledge distillation loss. This loss encourages the student MRI model's output to mimic the teacher TVUS model's output, even though the training data is unpaired. A regression loss is used to approximate the teacher's predictions.
  4. The final loss function combines this knowledge distillation loss with the original cross-entropy loss for the MRI classification task, allowing for a dynamic balance between the two during training.

Datasets

The study utilizes three datasets:

  • MRI Endometriosis Dataset: A private dataset containing 88 T2 SPACE MRI scans from women aged 18-45, with 19 cases of POD obliteration. This dataset is used for training and testing the MRI models.
  • TVUS Endometriosis Dataset: A private dataset of 749 TVUS video clips, including 103 negative (POD obliterated) cases. This dataset is used to train the teacher model.
  • Female Pelvic MRI Dataset: A large dataset of 8,984 MRI volumes used for self-supervised pre-training of the 3D MAE student model.

Key Insights and Ideas

  • Leveraging Unpaired Data: The most significant insight is the successful application of knowledge distillation to transfer information between modalities using unpaired datasets. This is crucial in medical imaging where obtaining perfectly matched multi-modal data is often impractical.
  • TVUS as a Superior Teacher: The paper validates the intuition that TVUS provides a more reliable signal for POD obliteration detection, making it an excellent "teacher" for a less capable MRI model.
  • MAE for Robust MRI Representation: Using 3D MAE for self-supervised pre-training on unlabeled MRI data is shown to be effective in learning generalizable features for the downstream task, mitigating the challenge of limited labeled MRI data.
  • Bridging the Domain Gap: The knowledge distillation process acts as a bridge, helping to align the feature spaces and decision boundaries of the TVUS and MRI models, thereby improving MRI performance.

Key Results

The proposed method, combining MAE pre-training, knowledge distillation, and fine-tuning (MAE PT + KD + FT), achieved a significant improvement in POD obliteration classification accuracy on MRI:

  • Baseline 3D ViT (MRI only): AUC of 65.0%.
  • 3D ViT with MAE Pre-training (MRI only): AUC improved to 87.2%.
  • Proposed Method (MAE PT + KD + FT): Achieved a final AUC of 90.6%, outperforming the baseline and the MAE pre-trained model without distillation.
  • For comparison, the TVUS teacher model achieved an impressive AUC of 96.9%.

Limitations and Future Work

  • Domain Shift: The authors acknowledge that despite the improvements, there's still a domain shift between the pre-training dataset (general pelvic MRIs) and the target endometriosis MRI dataset, which can sometimes hinder the KD performance.
  • Small Labeled MRI Dataset: The limited number of labeled MRI samples for endometriosis diagnosis remains a challenge.
  • Future Work: The authors plan to explore "missing modality deep learning approaches" and extend their method to weakly-supervised lesion segmentation for improved interpretability and broader clinical applicability.

Summary updated

Abstract

Endometriosis is a common chronic gynecological disorder that has many characteristics, including the pouch of Douglas (POD) obliteration, which can be diagnosed using Transvaginal gynecological ultrasound (TVUS) scans and magnetic resonance imaging (MRI). TVUS and MRI are complementary non-invasive endometriosis diagnosis imaging techniques, but patients are usually not scanned using both modalities and, it is generally more challenging to detect POD obliteration from MRI than TVUS. To mitigate this classification imbalance, we propose in this paper a knowledge distillation training algorithm to improve the POD obliteration detection from MRI by leveraging the detection results from unpaired TVUS data. More specifically, our algorithm pre-trains a teacher model to detect POD obliteration from TVUS data, and it also pre-trains a student model with 3D masked auto-encoder using a large amount of unlabelled pelvic 3D MRI volumes. Next, we distill the knowledge from the teacher TVUS POD obliteration detector to train the student MRI model by minimizing a regression loss that approximates the output of the student to the teacher using unpaired TVUS and MRI data. Experimental results on our endometriosis dataset containing TVUS and MRI data demonstrate the effectiveness of our method to improve the POD detection accuracy from MRI.
BibTeX
@inproceedings{zhang2023distilling,
  author = {Yuan Zhang and Hu Wang and David Butler and Minh-Son To and Jodie Avery and M. Louise Hull and Gustavo Carneiro},
  title = {{Distilling Missing Modality Knowledge from Ultrasound for Endometriosis Diagnosis with Magnetic Resonance Images}},
  booktitle = {ISBI},
  year = {2023},
  doi = {10.1109/isbi53787.2023.10230667},
}

Loading related content...

Figure