Spatio-Temporal Dual-Stream Neural Network for Sequential Whole-Body PET Segmentation

Kai-Chieh Liang, Lei Bi, Ashnil Kumar, Michael Fulham, Jinman KimarXiv 2021 · arXiv

TL;DR

Researchers have developed a Spatio-Temporal Dual-Stream Neural Network (ST-DSNN) that improves the automatic segmentation of whole-body PET scans by analyzing multiple scans taken over time. By leveraging the fact that healthy organs show consistent radioactive tracer uptake while tumors change with treatment, the model achieves superior accuracy in identifying anatomical structures, providing a more reliable baseline for monitoring lymphoma progression.

What Problem Does This Paper Address?

In the treatment of lymphoma, 18F-FDG PET scans are the gold standard for assessing how a patient is responding to therapy. These scans detect the metabolic activity of cells; however, they have a major "noise" problem. Several healthy organs—like the brain, heart, kidneys, and bladder—naturally accumulate the radioactive tracer (FDG). These areas of normal activity, termed sFEPU (sites of FDG excretion and physiologic uptake), can easily be confused with active tumor sites.

Current deep learning models for PET segmentation typically look at a single scan in isolation. This is a missed opportunity. Because sFEPU structures remain relatively consistent across sequential scans while diseased tissues change due to treatment, the temporal dimension holds the key to better differentiation. The challenge lies in designing an architecture that can effectively "remember" and compare these structures across different time points.

Methods

The authors propose the ST-DSNN, a dual-stream architecture designed to process two sequential PET scans (time $t$ and $t+1$) simultaneously.

Figure 2. Our proposed ST-DSNN method.

Key components include:

  • Weight-Sharing Encoders: Two identical CNN streams extract high-level features from each scan. Because they share weights, the model ensures that the feature maps are perfectly aligned in the latent space.
  • Feature Accumulation (Fusion): This is the "secret sauce." The model performs element-wise addition of the feature maps from both time points. Features representing consistent structures (like the heart or kidneys) are mathematically enhanced, while inconsistent features (like changing tumors or noise) are relatively suppressed.
  • Residual Generating Blocks: These blocks further refine the fused features to ensure the model focuses on temporally stable anatomical landmarks.
  • Dual Decoders: Finally, the model uses two decoders with skip connections to produce precise segmentation masks for both scans in the sequence.

Datasets

The study utilized a specialized dataset of sequential whole-body PET images from 10 lymphoma patients at the Royal Prince Alfred Hospital in Sydney.

  • Patient Breakdown: 6 patients had 2 scans, 3 had 3 scans, and 1 had 4 scans.
  • Image Pairing: To train the dual-stream network, the researchers generated 21 unique image pairs (e.g., Scan 1 paired with Scan 2, Scan 2 with Scan 3, etc.).
  • Ground Truth: Manual annotations were provided for five major sFEPU regions: brain, heart, bladder, and both kidneys.

Key Insights and Ideas

  • Temporal Consistency as a Filter: The core "aha moment" is treating anatomical organs as a stable signal and tumors as a transient one. By fusing features across time, the model essentially uses the patient's own anatomy from a previous scan to "verify" the anatomy in the current scan.
  • Balanced Learning: Standard models often overfit to "easy" dominant structures like the brain or bladder while failing on complex ones like the kidneys (which can show asymmetric excretion). The ST-DSNN's ability to accumulate features across scans helps balance the feature representation, leading to much better performance on these difficult organs.
  • Flexibility: While designed for sequences, the authors demonstrated that the model can still function with a single time point by feeding the same image into both streams, making it backwards-compatible with single-scan workflows.

Key Results

The ST-DSNN was compared against several heavyweights, including U-Net, VGG-FCN8s, and ConvLSTM.

  • Overall Accuracy: The proposed method achieved a Mean Dice Similarity Coefficient (DSC) of 90.03%, outperforming the second-best method (U-Net) by 1.8%.
  • Kidney Segmentation: The most dramatic improvements were seen in the kidneys, where the ST-DSNN outperformed U-Net by nearly 7-8% in DSC.
  • Temporal Advantage: Using sequential images improved the mean DSC by 0.9% compared to using the same image in both streams, proving that the temporal information is statistically significant for segmentation quality.
Table 1. Image pairs for our ST-DSNN.

Limitations and Future Work

The primary limitation is the small dataset size (10 patients), which is common in specialized medical imaging studies due to the difficulty of obtaining longitudinal data and expert annotations.

Looking ahead, the authors aim to:

  1. Test the architecture on other cancer types, such as non-small cell lung cancer and soft-tissue sarcomas.
  2. Expand the model to handle multi-modal data, integrating CT and MR images alongside PET scans to provide even more anatomical context.
  3. Evaluate how this improved sFEPU segmentation directly impacts the quantitative monitoring of tumor metabolic volume over time.

Summary updated

Abstract

Sequential whole-body 18F-Fluorodeoxyglucose (FDG) positron emission tomography (PET) scans are regarded as the imaging modality of choice for the assessment of treatment response in the lymphomas because they detect treatment response when there may not be changes on anatomical imaging. Any computerized analysis of lymphomas in whole-body PET requires automatic segmentation of the studies so that sites of disease can be quantitatively monitored over time. State-of-the-art PET image segmentation methods are based on convolutional neural networks (CNNs) given their ability to leverage annotated datasets to derive high-level features about the disease process. Such methods, however, focus on PET images from a single time-point and discard information from other scans or are targeted towards specific organs and cannot cater for the multiple structures in whole-body PET images. In this study, we propose a spatio-temporal 'dual-stream' neural network (ST-DSNN) to segment sequential whole-body PET scans. Our ST-DSNN learns and accumulates image features from the PET images done over time. The accumulated image features are used to enhance the organs / structures that are consistent over time to allow easier identification of sites of active lymphoma. Our results show that our method outperforms the state-of-the-art PET image segmentation methods.
BibTeX
@misc{liang2021spatio,
  author = {Kai-Chieh Liang and Lei Bi and Ashnil Kumar and Michael Fulham and Jinman Kim},
  title = {{Spatio-Temporal Dual-Stream Neural Network for Sequential Whole-Body PET Segmentation}},
  year = {2021},
  eprint = {2106.04961},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2106.04961},
}
Figure