Extreme Cardiac MRI Analysis under Respiratory Motion: Results of the CMRxMotion Challenge

Kang Wang, Chen Qin, Zhang Shi, Haoran Wang, Xiwen Zhang, Chen Chen, Cheng Ouyang, Chengliang Dai, Yuanhan Mo, Chenchen Dai, Xutong Kuang, Ruizhe Li, Xiuzheng Yue, Song Tian, Alejandro Mora-Rubio, Kumaradevan Punithakumar, Shizhan Gong, Qi Dou, Sina Amirrajab, Yasmina Al Khalil, Cian M. Scannell, Lexiaozi Fan, Huili Yang, Xiaowu Sun, Rob van der Geest, Tewodros Weldebirhan Arega, Fabrice Meriaudeau, Caner Özer, Amin Ranem, John Kalkhof, İlkay Öksüz, Anirban Mukhopadhyay, Abdul Qayyum, Moona Mazher, Steven A Niederer, Carles Garcia-Cabrera, Eric Arazo, Michal K. Grzeszczyk, Szymon Płotka, Wanqin Ma, Xiaomeng Li, Rongjun Ge, Yongqing Kou, He Wang, Chengyan Wang, Wenjia Bai, Shuo Wang

TL;DR

Deep learning models excel at cardiac MRI analysis but frequently degrade when images are blurred by patient breathing. The CMRxMotion challenge introduces a newly curated dataset with controlled respiratory motion artifacts to benchmark automated image quality assessment and robust myocardial segmentation. The findings show that while AI approaches human-level performance on clean scans, respiratory motion significantly compromises both segmentation accuracy and downstream clinical biomarkers.

What Problem Does This Paper Address?

Automated Cardiac Magnetic Resonance (CMR) image analysis relies heavily on high-quality, artifact-free images. In clinical settings, however, patients—especially pediatric patients or those suffering from heart failure—often struggle to follow strict breath-holding instructions. This results in respiratory motion artifacts that blur anatomical boundaries and corrupt deep learning model predictions.

While previous medical imaging benchmarks have focused extensively on scanner vendor variability or anatomical diversity, the impact of unexpected human breathing behaviors has remained largely unaddressed. This paper summarizes the design and results of the MICCAI CMRxMotion challenge, which explicitly tasks machine learning models with evaluating image quality and performing robust segmentation under varying degrees of respiratory motion.

Methods

The challenge was divided into two distinct tasks:

  1. Automated Image Quality Assessment (IQA): Teams developed models to classify 3D CMR volumes into three categories: mild, intermediate, or severe motion. The winning approach (A1. UON_IMA) leveraged a hierarchical ensemble of 2D CNNs (ResNet, EfficientNet) and Vision Transformers (ViT) pre-trained on ImageNet, utilizing both raw intensity images and 2D in-plane gradient magnitude maps.
  2. Robust Cardiac Segmentation (RCS): Algorithms delineated the left ventricle (LV), right ventricle (RV), and myocardium (MYO) in the presence of motion. The top spot resulted in a tie between S1. UA-SVCC (a modified nnU-Net using a multi-stage training progression from cross-entropy to Dice loss) and S2. Med-Air (an ensemble of nnU-Net, Swin-UNETR, and Swin-UNet utilizing adversarial data augmentation via the AdvChain framework).

Datasets

The authors prospectively curated a unique dataset of 320 3D CMR cine volumes from 40 healthy volunteers. To isolate respiratory motion from confounding factors like scanner vendor or underlying pathology, each volunteer underwent four separate scans on the same 3T scanner during a single visit. Each scan used a different breathing protocol: standard breath-hold, half breath-hold, free breathing, and intensive breathing.

Figure 1: Visualization example of paired short-axis (SAX) CMR images with different Image Quality Assessment (IQA) scores, acquired from the same volunteer performing four distinct breath-hold behaviors following specific breathing guidance during CMR acquisition. (b) End-diastolic (ED) images, showing three images with progressively increasing IQA scores, are presented in the top row. (c) End-systolic (ES) images, which also display three CMR scans in both axial and coronal planes, are presented in the bottom row.

Slices were extracted at the end-diastolic (ED) and end-systolic (ES) phases. The dataset was partitioned at the subject level into a training set (160 volumes), a validation set (40 volumes), and a fully withheld test set (120 volumes).

Key Insights and Ideas

  • 2D and Lightweight Models Outperformed 3D: Interestingly, 2D slice-wise models generally outperformed their 3D counterparts, and lightweight backbones proved highly competitive. This is likely because smaller models face a reduced risk of overfitting on relatively small medical datasets.
  • The Intermediate-to-Severe Blindspot: While models were highly accurate at identifying pristine, high-quality images (mild motion), they struggled significantly to differentiate between intermediate and severe motion artifacts. Fine-grained artifact assessment remains an open research problem.
  • The Downstream Domino Effect: Subtle segmentation errors caused by motion artifacts don't just lower validation scores; they exponentially amplify errors in critical clinical biomarkers like ejection fraction and ventricular volumes, which could directly impact patient management and diagnosis.

Key Results

  • Quality Assessment: For the IQA task, the winning team achieved a Cohen’s Kappa ($\kappa$) of 0.631 and an overall accuracy of 0.725.
  • Segmentation Accuracy: For the segmentation task, the top models achieved impressive average Dice scores across the full test set: up to 93.90% for the LV, 92.39% for the RV, and 87.38% for the complex, thin-walled myocardium.
Figure 2: Visualization of segmentation labels for Task 2. (a) Volume ren- dering of a short-axis CMR image. (b) Volume rendering of the correspond- ing segmentation mask. (c-h) Paired image slices and segmentation masks for the basal, mid-ventricular, and apical regions. The LV is shown in red, MYO in green, and RV in blue.
  • Impact of Motion: Subgroup analysis revealed that as motion severity increased from mild to severe, the top algorithm's LV segmentation Dice score dropped by over 3 percentage points, while the Mean Absolute Error (MAE) for the derived Left Ventricular End-Diastolic Volume (LVEDV) more than doubled.

Limitations and Future Work

  • Dataset Homogeneity: The dataset was collected exclusively from healthy volunteers at a single imaging center, meaning it lacks real-world clinical diversity and pathological variations.
  • Computational Overhead: The most accurate segmentation models relied heavily on massive, multi-model ensembles. This makes them computationally intensive and slow—a major hurdle for real-time clinical deployment.
  • Future Directions: The authors suggest exploring Vision Foundation Models, optimizing lightweight architectures for rapid inference, and developing synergistic frameworks that combine motion-robust segmentation directly with accelerated, motion-corrected MRI reconstruction at the source.

Summary updated

Abstract

Deep learning models have achieved state-of-the-art performance in automated Cardiac Magnetic Resonance (CMR) analysis. However, the efficacy of these models is highly dependent on the availability of high-quality, artifact-free images. In clinical practice, CMR acquisitions are frequently degraded by respiratory motion, yet the robustness of deep learning models against such artifacts remains an underexplored problem. To promote research in this domain, we organized the MICCAI CMRxMotion challenge. We curated and publicly released a dataset of 320 CMR cine series from 40 healthy volunteers who performed specific breathing protocols to induce a controlled spectrum of motion artifacts. The challenge comprised two tasks: 1) automated image quality assessment to classify images based on motion severity, and 2) robust myocardial segmentation in the presence of motion artifacts. A total of 22 algorithms were submitted and evaluated on the two designated tasks. This paper presents a comprehensive overview of the challenge design and dataset, reports the evaluation results for the top-performing methods, and further investigates the impact of motion artifacts on five clinically relevant biomarkers. All resources and code are publicly available at: https://github.com/CMRxMotion
BibTeX
@article{wang2025extreme,
  author = {Kang Wang and Chen Qin and Zhang Shi and Haoran Wang and Xiwen Zhang and Chen Chen and Cheng Ouyang and Chengliang Dai and Yuanhan Mo and Chenchen Dai and Xutong Kuang and Ruizhe Li and Xin Chen and Xiuzheng Yue and Song Tian and Alejandro Mora-Rubio and Kumaradevan Punithakumar and Shizhan Gong and Qi Dou and Sina Amirrajab and Yasmina Al Khalil and Cian M. Scannell and Lexiaozi Fan and Huili Yang and Xiaowu Sun and Rob van der Geest and Tewodros Weldebirhan Arega and Fabrice Meriaudeau and Caner Özer and Amin Ranem and John Kalkhof and İlkay Öksüz and Anirban Mukhopadhyay and Abdul Qayyum and Moona Mazher and Steven A Niederer and Carles Garcia-Cabrera and Eric Arazo and Michal K. Grzeszczyk and Szymon Płotka and Wanqin Ma and Xiaomeng Li and Rongjun Ge and Yongqing Kou and Xinrong Chen and He Wang and Chengyan Wang and Wenjia Bai and Shuo Wang},
  title = {{Extreme Cardiac MRI Analysis under Respiratory Motion: Results of the CMRxMotion Challenge}},
  journal = {Medical Image Analysis},
  year = {2025},
}
Figure