Uncertainty-based Quality Control for Subcortical Structures Segmentation in T1-weighted Brain MRI

Benjamin Lambert, Florence Forbes, Senan Doyle, Alan Tucholka, Michel DojatProceedings on CD-ROM - International Society for Magnetic Resonance in Medicine. Scientific Meeting and Exhibition/Proceedings of the International Society for Magnetic Resonance in Medicine, Scientific Meeting and Exhibition 2024 · DOI

TL;DR

This paper tackles the critical problem of "silent failures" in deep learning models used for brain MRI segmentation. They demonstrate that a method called Deterministic Uncertainty Method (DUM) is significantly better than a simpler approach (Maximum Softmax Probability) at identifying MRI scans with artifacts or characteristics not seen during training, ensuring more reliable clinical applications.

What Problem Does This Paper Address?

Deep learning (DL) models have become the go-to for medical image segmentation, including identifying structures in brain MRIs. However, these models can be brittle. If a new MRI scan has characteristics (like noise, artifacts, or different imaging parameters) that weren't present in the data the model was trained on, its performance can plummet without any warning. This is known as an "Out-of-Distribution" (OOD) problem, and it's a major hurdle for deploying these powerful tools in real-world clinical settings where data variability is high. The authors focus on this issue within the context of segmenting subcortical structures (like the hippocampus and thalamus) from T1-weighted brain MRIs.

Methods

The researchers compare two methods for detecting these OOD inputs:

  • Maximum Softmax Probability (MSP): This is a straightforward approach. It assumes that for "in-distribution" (ID) images (those similar to the training data), the DL model will be very confident in its predictions, assigning high probabilities to the correct segmentation class. OOD images, on the other hand, are expected to result in lower confidence scores. They calculate the average MSP across the entire MRI volume as a single score for each image.
  • Deterministic Uncertainty Method (DUM): This is a more sophisticated technique. It leverages the intermediate activations within the DL model (specifically, from the penultimate layer of an Attention U-Net). They compute the Singular Value Decomposition (SVD) of these activations to create a "spectral signature" for each image. The conformity score is then determined by the distance between the test image's spectral signature and the signatures of the training images. A larger distance suggests an OOD input.

Datasets

  • IXI Dataset: This dataset, comprising 581 healthy T1-weighted brain MRIs, was used for training and evaluating the segmentation models.
  • Synthetic OOD Data: To simulate real-world artifacts, the authors generated 7 types of OOD images from the original IXI scans using the TorchiO library. These included Downsample, Bias, Spikes, Ghost, Motion, Gaussian Noise, and Scale artifacts. These synthetic OOD images are representative of common issues encountered in clinical MRI.

Key Insights and Ideas

  • The "silent failure" is a real threat: The paper strongly emphasizes that DL models can fail dramatically on OOD data without any obvious indication, posing a significant risk.
  • DUM offers a more robust OOD signal: Unlike MSP, which can be fooled by certain artifacts, DUM's reliance on internal model representations provides a more stable and reliable indicator of OOD data.
  • Artifacts have varying impacts: Some artifacts (like Spikes and Gaussian Noise) severely degrade segmentation quality, making OOD detection crucial. Others have a less pronounced effect on segmentation but can still be detected by DUM.
  • Generalizability of DUM: The success of DUM in this specific segmentation task suggests its potential applicability to other medical imaging domains and DL tasks.

Key Results

  • Segmentation Quality: Dice Scores (DSC) generally decreased for OOD images compared to in-distribution images. However, this drop was particularly severe for Spikes and Gaussian Noise artifacts.
  • OOD Detection Performance (AUROC):
    • For Spikes and Gaussian Noise artifacts, DUM achieved a perfect AUROC score of 1.0, meaning it could perfectly distinguish between ID and OOD images. MSP performed poorly on these artifacts.
  • Overall, DUM surpassed MSP on 6 out of 7 OOD artifact types for both hippocampus and thalamus segmentation.
  • The paper highlights that DUM allows for robust identification of non-conform MRIs, preventing potentially poor decisions based on unreliable segmentation.

Limitations and Future Work

  • Synthetic Artifacts: While the generated artifacts are representative, they are synthetic. Future work could involve evaluating DUM on real-world, naturally occurring artifacts.
  • Specific Structures: The study focused on hippocampus and thalamus. Further validation on a wider range of anatomical structures and different MRI sequences would be beneficial.
  • Computational Cost: While DUM is described as computationally efficient, the paper doesn't provide specific benchmarks compared to MSP or other OOD detection methods.

Summary updated

Abstract

The performance of Deep Learning (DL) models may drastically drop in the presence of characteristics in test images not present in the training set. Then, the automatic detection of these Out-Of-Distribution (OOD) inputs is important to deploy these methods especially for clinical applications. We address this issue in the context of DL-based subcortical structures segmentation on T1w brain MRI. We compare two OOD detection frameworks equipping DL segmentation models, Maximum Softmax Probability and Deterministic Uncertainty Method, and demonstrate the superiority of the latter which allows a robust and versatile identification of artifacts in images.
BibTeX
@inproceedings{lambert2024uncertainty,
  author = {Benjamin Lambert and Florence Forbes and Senan Doyle and Alan Tucholka and Michel Dojat},
  title = {{Uncertainty-based Quality Control for Subcortical Structures Segmentation in T1-weighted Brain MRI}},
  booktitle = {Proceedings on CD-ROM - International Society for Magnetic Resonance in Medicine. Scientific Meeting and Exhibition/Proceedings of the International Society for Magnetic Resonance in Medicine, Scientific Meeting and Exhibition},
  year = {2024},
  doi = {10.58530/2023/3309},
}
Figure