MPBD-LSTM: A Predictive Model for Colorectal Liver Metastases Using Time Series Multi-phase Contrast-Enhanced CT Scans

Xueyang Li, Han Xiao, Weixiang Weng, Xiaowei Xu, Yiyu Shi

Introduction

Colorectal cancer is a leading cause of cancer-related death, frequently spreading to the liver. Early detection of colorectal liver metastases (CRLM) is the "holy grail" for improving patient survival. In clinical practice, radiologists monitor patients using Contrast-Enhanced CT (CECT) scans during longitudinal follow-up visits. This creates a massive, five-dimensional data challenge:

  1. Spatial (x, y, z): The 3D volume of the liver.
  2. Temporal: Scans taken at different follow-up visits over months or years.
  3. Phase: Different contrast phases (Arterial vs. Venous) based on when contrast dye hits the vasculature.

Most existing AI models are designed for 4D data (like video or 3D volumes over time). When faced with the 5th dimension—the contrast phase—standard models usually concatenate the data, which can lead to a loss of spatial alignment and vital diagnostic nuances.

Methods: The MPBD-LSTM Architecture

To address these challenges, researchers developed the Multi-Plane Bi-Directional LSTM (MPBD-LSTM). It builds upon the E3D-LSTM architecture but introduces two critical modifications to handle 5D medical data natively:

  • Multi-Plane Architecture: Instead of merging contrast phases into a single input channel, the model uses two parallel "planes" (Arterial and Portal Venous). This allows the model to extract features from each phase independently before fusion, preserving the unique vascular patterns of each.
  • Bi-Directional Temporal Modeling: Standard LSTMs process sequences chronologically. MPBD-LSTM uses bi-directional connections, allowing the model to "look" both forward and backward across a patient’s history of scans. This helps the network better understand how subtle changes in liver tissue evolve over time.
The MPBD-LSTM architecture utilizes parallel planes for different contrast phases and bi-directional connections for temporal modeling.

Dataset and Evaluation

The study utilized a retrospective dataset consisting of 269 patients (69 positive for CRLM, 200 negative) across two hospital cohorts. Each patient had between 2 and 6 follow-up CECT scans. Scans were normalized to a consistent 64-slice depth using Spline Interpolated Zoom (SIZ). Data augmentation techniques, including Standard Scale Jittering (SSJ) and Mixup, were applied to address class imbalance and improve model robustness.

Key Results and Insights

MPBD-LSTM achieved a state-of-the-art AUC of 0.790, significantly outperforming standard spatiotemporal models.

Performance comparison of MPBD-LSTM against baseline and state-of-the-art spatiotemporal models.

Key insights from the study include:

  • Parallelism vs. Concatenation: Ablation studies showed that processing phases in parallel (multi-plane) is superior to simple concatenation (AUC 0.790 vs 0.774). Parallel processing keeps distinct anatomical "views" separate, preventing model confusion.
  • Phase Importance: The model confirmed clinical intuition: the Portal Venous phase is more informative (AUC 0.752) than the Arterial phase (AUC 0.653), though the multi-phase approach remains optimal.
  • Temporal Context: Bi-directional modeling improved performance (AUC 0.790 vs 0.768), suggesting that later scans provide valuable context for interpreting ambiguous findings in earlier scans.

Conclusion

MPBD-LSTM demonstrates the power of specialized architectures for high-dimensional medical imaging. While the model shows great promise, future work will focus on addressing anatomical variability—such as liver size—and expanding the dataset to improve clinical generalization and move toward autonomous clinical application.

Summary updated

BibTeX
@inproceedings{li2023mpbd,
  author = {Xueyang Li and Han Xiao and Weixiang Weng and Xiaowei Xu and Yiyu Shi},
  title = {{MPBD-LSTM: A Predictive Model for Colorectal Liver Metastases Using Time Series Multi-phase Contrast-Enhanced CT Scans}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2023},
  url = {https://openreview.net/forum?id=dVhxUjveaW},
}
Figure