Spatial-Temporal Convolutional LSTMs for Tumor Growth Prediction by Learning 4D Longitudinal Patient Data

Ling Zhang, Le Lu, Robert Zhu, Mohammadhadi Bagheri, Ronald M. Summers, Jianhua YaoarXiv 2019 · arXiv

Abstract

Predicting tumor progression is a critical challenge in personalized oncology. Traditional methods often rely on simplistic mathematical models or 2D deep learning architectures that fail to capture the full 3D spatial and temporal complexity of longitudinal medical imaging. This paper introduces the Spatio-Temporal Convolutional LSTM (ST-ConvLSTM), an end-to-end framework that jointly learns 3D spatial context and temporal dynamics from 4D patient data. By treating 3D volumes as sequences of slices and incorporating multi-channel inputs (CT intensity, cell density, and tumor masks), the model predicts future tumor states with high accuracy. Experiments on a longitudinal Pancreatic Neuroendocrine Tumor (PanNET) dataset demonstrate that ST-ConvLSTM achieves a Dice score of 83.2% for volume prediction, outperforming existing baselines and providing a holistic view of tumor evolution.

Introduction

Personalized medicine for cancer patients often hinges on the ability to predict tumor growth. Clinicians must decide whether a tumor requires immediate intervention or can be safely monitored through "active surveillance." However, modeling tumor progression is a difficult "4D" problem involving 3D spatial structures evolving over time.

Existing approaches typically fall into two categories:

  1. Mathematical Models: Biologically grounded equations (e.g., reaction-diffusion) that often lack the parameter complexity to capture real-world tumor heterogeneity.
  2. 2D Deep Learning: Data-driven models that often process 2D image patches, ignoring the 3D anatomical context and failing to predict clinically relevant properties beyond simple volume, such as cell density or tissue intensity changes.

Methodology

The authors propose an end-to-end ST-ConvLSTM architecture designed to overcome these limitations by processing 3D volumes as sequences of 2D slices over time.

Data Preprocessing

To prepare the longitudinal data, the authors developed a pipeline involving image registration, tumor segmentation, and the calculation of the Intracellular Volume Fraction (ICVF), which represents cell density.

Overview of the data preprocessing pipeline, including registration, segmentation, and ICVF calculation.

ST-ConvLSTM Architecture

The core of the framework is the ST-ConvLSTM unit. Unlike standard ConvLSTMs designed for 2D video, the ST-ConvLSTM handles 3D volumes by receiving hidden states from two directions:

  • Horizontal: Temporal information from the same slice location at a previous time point.
  • Vertical: Spatial information from the adjacent slice at the current time point.

This dual-flow mechanism ensures spatial consistency across the Z-axis while tracking temporal changes.

Multi-Channel and Clinical Integration

The model processes a three-channel input: ICVF (cell density), CT Intensity (soft-tissue windowed), and a binary Tumor Mask. Additionally, non-imaging clinical factors, such as the time interval between scans, are integrated by tiling numerical values into feature maps and concatenating them with image features.

Experiments and Results

The model was primarily evaluated on the PanNET Dataset, comprising longitudinal dual-phase contrast-enhanced CT scans from 33 patients with Pancreatic Neuroendocrine Tumors. To test generalizability, it was also applied to the CETUS Dataset for 4D ultrasound segmentation of the left ventricle.

Key Performance Metrics

  • Volume Prediction: The ST-ConvLSTM achieved a Dice score of 83.2% and a Relative Volume Difference (RVD) of 11.2%, significantly outperforming linear models, standard ConvLSTMs, and GAN-based approaches.
  • Holistic Prediction: The model successfully predicted future cell density (RMSE of 14.0%) and CT radiodensity, providing a more comprehensive view of tumor progression.
  • Clinical Utility: The model demonstrated 76.2% sensitivity in predicting whether a tumor would progress or regress.
  • Efficiency: Once trained, the model can predict a future tumor state in just 0.2 seconds, offering a massive speed advantage over traditional mathematical simulations.

Discussion and Conclusion

While the ST-ConvLSTM shows state-of-the-art performance, some limitations remain. Predicted CT images can appear blurry due to the $\ell_2$ (Mean Squared Error) loss function, and the model occasionally struggles with extreme outliers—tumors that grow or shrink at exceptional rates. Future work will focus on incorporating adversarial training to sharpen predictions and expanding the dataset to improve robustness.

In conclusion, the ST-ConvLSTM framework represents a significant step forward in 4D medical imaging. By jointly modeling spatial and temporal dependencies, it provides a powerful tool for predicting tumor evolution and supporting personalized clinical decision-making.

Summary updated

Abstract

Prognostic tumor growth modeling via volumetric medical imaging observations can potentially lead to better outcomes of tumor treatment and surgical planning. Recent advances of convolutional networks have demonstrated higher accuracy than traditional mathematical models in predicting future tumor volumes. This indicates that deep learning-based techniques may have great potentials on addressing such problem. However, current 2D patch-based modeling approaches cannot make full use of the spatio-temporal imaging context of the tumor's longitudinal 4D (3D + time) data. Moreover, they are incapable to predict clinically-relevant tumor properties, other than volumes. In this paper, we exploit to formulate the tumor growth process through convolutional Long Short-Term Memory (ConvLSTM) that extract tumor's static imaging appearances and capture its temporal dynamic changes within a single network. We extend ConvLSTM into the spatio-temporal domain (ST-ConvLSTM) by jointly learning the inter-slice 3D contexts and the longitudinal or temporal dynamics from multiple patient studies. Our approach can incorporate other non-imaging patient information in an end-to-end trainable manner. Experiments are conducted on the largest 4D longitudinal tumor dataset of 33 patients to date. Results validate that the ST-ConvLSTM produces a Dice score of 83.2%+-5.1% and a RVD of 11.2%+-10.8%, both significantly outperforming (p<0.05) other compared methods of linear model, ConvLSTM, and generative adversarial network (GAN) under the metric of predicting future tumor volumes. Additionally, our new method enables the prediction of both cell density and CT intensity numbers. Last, we demonstrate the generalizability of ST-ConvLSTM by employing it in 4D medical image segmentation task, which achieves an averaged Dice score of 86.3+-1.2% for left-ventricle segmentation in 4D ultrasound with 3 seconds per patient.
BibTeX
@misc{zhang2019spatial,
  author = {Ling Zhang and Le Lu and Robert Zhu and Mohammadhadi Bagheri and Ronald M. Summers and Jianhua Yao},
  title = {{Spatial-Temporal Convolutional LSTMs for Tumor Growth Prediction by Learning 4D Longitudinal Patient Data}},
  year = {2019},
  eprint = {1902.08716},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1902.08716},
}
Figure