MPBD-LSTM: A Predictive Model for Colorectal Liver Metastases Using Time Series Multi-phase Contrast-Enhanced CT Scans

Xueyang Li, Han Xiao, Weixiang Weng, Xiaowei Xu, Yiyu Shi

TL;DR

Researchers have developed MPBD-LSTM, a specialized deep learning model designed to predict the onset of colorectal liver metastases (CRLM) by analyzing "5D" medical imaging data. By utilizing a multi-plane architecture to process different contrast phases in parallel and bi-directional LSTMs to capture long-term patient history, the model achieves a state-of-the-art AUC of 0.79, significantly outperforming standard spatiotemporal models.

What Problem Does This Paper Address?

Colorectal cancer is a leading cause of cancer-related death, often because it spreads to the liver. Early detection of these metastases (CRLM) is the "holy grail" for improving patient survival.

In clinical practice, radiologists monitor patients using Contrast-Enhanced CT (CECT) scans during follow-up visits. This creates a massive, complex data challenge. The data is effectively five-dimensional:

  1. Spatial (x, y, z): The 3D volume of the liver.
  2. Temporal: Scans taken at different follow-up visits over months or years.
  3. Phase: Different "looks" at the liver based on when contrast dye hits the arteries versus the veins.

Most existing AI models are designed for 4D data (like video or 3D volumes over time). When faced with the 5th dimension—the contrast phase—standard models usually just "squash" or concatenate the data, which can lead to a loss of spatial alignment and vital diagnostic nuances. This paper seeks a way to handle all five dimensions natively.

Methods

The authors introduce the Multi-Plane Bi-Directional LSTM (MPBD-LSTM). It builds upon the E3D-LSTM (Eidetic 3D LSTM) architecture but introduces two critical modifications:

  • Multi-Plane Architecture: Instead of shoving all contrast phases into one input channel, the model uses two parallel "planes" (one for the Arterial phase and one for the Portal Venous phase). This allows the model to extract features from each phase independently before fusing them, preserving the unique vascular patterns of each.
  • Bi-Directional Temporal Modeling: Standard LSTMs process sequences in one direction (past to present). MPBD-LSTM uses bi-directional connections, allowing the model to "look" both forward and backward across a patient’s history of scans. This helps the network better understand how subtle changes in liver tissue evolve over time.
  • 3D-CNN Encoders: Before the data hits the LSTM layers, 3D Convolutional Neural Networks extract high-level spatial features from the CT volumes.

Datasets

The study utilized a private, retrospective dataset consisting of two cohorts from different hospitals:

  • Total Cases: 269 patients (69 positive for CRLM, 200 negative).
  • Data Structure: Each patient had between 2 and 6 follow-up CECT scans. Each scan included three phases (Plain, Arterial, and Venous) with 100–200 slices per volume.
  • Preprocessing: The researchers used Spline Interpolated Zoom (SIZ) to normalize all scans to a consistent 64-slice depth and applied data augmentation techniques like Standard Scale Jittering (SSJ) and Mixup to handle the class imbalance (more negative cases than positive).

Key Insights and Ideas

  • Parallelism Trumps Concatenation: The "Aha!" moment in the ablation study was discovering that processing phases in parallel (multi-plane) is superior to simple concatenation. Concatenation confuses the model because the same anatomical location looks different across phases; parallel processing keeps these "views" distinct.
  • The Importance of the Venous Phase: The model confirmed clinical intuition: the Portal Venous (V) phase is much more informative for predicting metastasis (AUC 0.752) than the Arterial (A) phase (AUC 0.653). However, using both together is still the best approach.
  • The "Future" Informs the "Past": By using bi-directional LSTMs, the model can use information from a later scan to better interpret ambiguous spots in an earlier scan, leading to a more robust representation of the disease's progression.

Key Results

MPBD-LSTM achieved an AUC of 0.790, the highest among all tested models. For comparison:

  • PredRNN-V2: 0.765
  • E3D-LSTM (Baseline): 0.755
  • SaConvLSTM: 0.721
  • SimVP (CNN-based): 0.662

The ablation studies showed that removing the multi-plane structure dropped the AUC to 0.774, and removing the bi-directional connections dropped it to 0.768, proving that both innovations are necessary for peak performance.

Limitations and Future Work

  • Dataset Size: While 269 cases is significant for this type of specialized 5D data, it is still small for deep learning. Larger, multi-center datasets are needed to improve generalization.
  • Anatomical Variability: Error analysis revealed the model struggles with patients who have smaller-than-average livers. The authors suggest that inter-patient variability in liver size and shape is a major hurdle that future models must address.
  • Performance Gap: While 0.79 is a strong result, the authors acknowledge there is still "great room for further improvement" before such a system could be used autonomously in a clinical setting. Future work will likely focus on more sophisticated ways to fuse 5D features.

Summary updated

Abstract

Colorectal cancer is a prevalent form of cancer, and many patients develop colorectal cancer liver metastasis (CRLM) as a result. Early detection of CRLM is critical for improving survival rates. Radiologists usually rely on a series of multi-phase contrast-enhanced computed tomography (CECT) scans done during follow-up visits to perform early detection of the potential CRLM. These scans form unique five-dimensional data (time, phase, and axial, sagittal, and coronal planes in 3D CT). Most of the existing deep learning models can readily handle four-dimensional data (e.g., time-series 3D CT images) and it is not clear how well they can be extended to handle the additional dimension of phase. In this paper, we build a dataset of time-series CECT scans to aid in the early diagnosis of CRLM, and build upon state-of-the-art deep learning techniques to evaluate how to best predict CRLM. Our experimental results show that a multi-plane architecture based on 3D bi-directional LSTM, which we call MPBD-LSTM, works best, achieving an area under curve (AUC) of 0.79. On the other hand, analysis of the results shows that there is still great room for further improvement. Our code is available at https://github.com/XueyangLiOSU/MPBD-LSTM.
BibTeX
@inproceedings{li2023mpbd,
  author = {Xueyang Li and Han Xiao and Weixiang Weng and Xiaowei Xu and Yiyu Shi},
  title = {{MPBD-LSTM: A Predictive Model for Colorectal Liver Metastases Using Time Series Multi-phase Contrast-Enhanced CT Scans}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2023},
  url = {https://openreview.net/forum?id=dVhxUjveaW},
}

Loading related content...

Figure