A task-unified network with transformer and spatial–temporal convolution for left ventricular quantification

Dapeng Li, Yanjun Peng, Jindong Sun, Yanfei GuoScientific 2023 · DOI

TL;DR

Researchers have developed a "task-unified" deep learning framework that simultaneously segments the left ventricle (LV) and calculates key clinical metrics from cardiac MRI. By combining a 3D Transformer for anatomical mapping with spatial-temporal convolutions for metric regression, the model achieves state-of-the-art accuracy while providing clinicians with the "visual proof" often missing in "black-box" AI models.

What Problem Does This Paper Address?

Quantifying the function of the left ventricle—the heart's main pumping chamber—is the gold standard for diagnosing cardiovascular diseases. Clinicians look at metrics like ejection fraction, wall thickness, and cavity area. Traditionally, this is done in one of two ways:

  1. Indirect (Segmentation-based): An AI outlines the heart, and then math is used to calculate metrics. This is interpretable but prone to error propagation; if the outline is slightly off, the metrics fail.
  2. Direct (Regression-based): An AI looks at the image and spits out numbers directly. This is often more accurate but provides no visual context, making it a "black box" that doctors are hesitant to trust.

Furthermore, the heart is a moving target. Its anatomy changes drastically between its relaxed (diastole) and contracted (systole) states. Most existing models struggle to capture this temporal "rhythm" effectively while maintaining high spatial precision.

Methods

The authors propose a dual-path, end-to-end architecture that treats segmentation and quantification as a single, unified task:

  • The Segmentation Path (3D TransUNet): This module uses a hybrid CNN-Transformer architecture. While standard CNNs are great at local details, Transformers excel at capturing "long-range dependencies"—essentially understanding how the whole heart fits together. By using 3D convolutions, the model processes five consecutive MRI slices at once, allowing it to learn the heart's motion over time.
  • The Regression Path: This path estimates the actual numbers (area, dimensions, wall thickness). It utilizes Recurrent Residual Attention Convolutional (RRAconv) blocks to focus on the most relevant parts of the image.
  • The "Unified" Secret Sauce: The model doesn't run these paths in isolation. Feature maps from the segmentation path are fed into the regression path via skip connections. This ensures the regression module "sees" the cleaned-up anatomical structures, reducing noise and improving numerical accuracy.
  • Joint Task Loss: The network is trained using a complex loss function that penalizes errors in both the visual outlines and the final clinical numbers simultaneously.

Datasets

The framework was trained and evaluated on two major benchmarks:

  • MICCAI 2017 LV-Quan Challenge: A dataset of 2,900 cardiac MRI images from 145 patients, featuring diverse pathologies like myocardial hypertrophy and enlarged LVs.
  • MICCAI 2009 Sunnybrook Cardiac Dataset: Used specifically to validate the segmentation performance against classic models.

Key Insights and Ideas

  • Anatomy as a Regularizer: By forcing the model to segment the heart, the researchers essentially "teach" the model anatomy. This anatomical knowledge makes the direct regression of metrics much more robust than if the model were just looking for patterns in raw pixels.
  • Transformers for Variability: The heart's shape varies wildly between patients and across the cardiac cycle. The Transformer's ability to model global context makes it significantly better at handling this variability than traditional U-Nets.
  • Temporal Dynamics: Treating the MRI as a 3D volume (2D + time) rather than a series of static 2D images is crucial. The neighbor slices provide the context needed to resolve blurry boundaries in low-contrast areas.

Key Results

The model achieved highly competitive results across the board:

  • Segmentation Accuracy: It reached a Dice score of 0.971 for the LV cavity and 0.921 for the myocardium, outperforming standard models like UNet and Densenet.
  • Quantification Precision: The model achieved a Mean Absolute Error (MAE) of just 132 $mm^2$ for area and 1.16 mm for regional wall thickness.
  • Clinical Correlation: The Pearson Correlation Coefficient (PCC) for area and dimensions was above 0.96, indicating an extremely high level of agreement with expert radiologist annotations.
  • Efficiency: Despite its complexity, the model maintained a relatively small GPU memory footprint (1.11G) and fast testing times, making it potentially viable for real-time clinical use.

Limitations and Future Work

While the results are impressive, the authors note a few areas for growth:

  • Dataset Diversity: While the MICCAI datasets are standard, the authors aim to verify the framework on even larger, more diverse real-world clinical datasets to ensure generalizability.
  • Full Heart Analysis: This study focused specifically on the left ventricle. Future iterations could expand the "task-unified" approach to include the right ventricle and the atria for a holistic view of cardiac health.
  • Complexity: The model has significantly more parameters (33.10M) than a basic UNet (1.43M), which might require more specialized hardware for training, though inference remains efficient.

Summary updated

Abstract

Quantification of the cardiac function is vital for diagnosing and curing the cardiovascular diseases. Left ventricular function measurement is the most commonly used measure to evaluate the function of cardiac in clinical practice, how to improve the accuracy of left ventricular quantitative assessment results has always been the subject of research by medical researchers. Although considerable efforts have been put forward to measure the left ventricle (LV) automatically using deep learning methods, the accurate quantification is yet a challenge work as a result of the changeable anatomy structure of heart in the systolic diastolic cycle. Besides, most methods used direct regression method which lacks of visual based analysis. In this work, a deep learning segmentation and regression task-unified network with transformer and spatial-temporal convolution is proposed to segment and quantify the LV simultaneously. The segmentation module leverages a U-Net like 3D Transformer model to predict the contour of three anatomy structures, while the regression module learns spatial-temporal representations from the original images and the reconstruct feature map from segmentation path to estimate the finally desired quantification metrics. Furthermore, we employ a joint task loss function to train the two module networks. Our framework is evaluated on the MICCAI 2017 Left Ventricle Full Quantification Challenge dataset. The results of experiments demonstrate the effectiveness of our framework, which achieves competitive cardiac quantification metric results and at the same time produces visualized segmentation results that are conducive to later analysis.
BibTeX
@inproceedings{li2023task,
  author = {Dapeng Li and Yanjun Peng and Jindong Sun and Yanfei Guo},
  title = {{A task-unified network with transformer and spatial–temporal convolution for left ventricular quantification}},
  booktitle = {Scientific},
  year = {2023},
  doi = {10.1038/s41598-023-40841-y},
}
Figure