Multi-phase Liver Tumor Segmentation with Spatial Aggregation and Uncertain Region Inpainting

Yue Zhang, Chengtao Peng, Liying Peng, Huimin Huang, Ruofeng Tong, Lanfen Lin, Jingsong Li, Yen-Wei Chen, Qingqing Chen, Hongjie Hu, Zhiyi Peng

TL;DR

This paper introduces a novel deep learning approach for segmenting liver tumors in multi-phase CT scans. It addresses the limitations of existing methods by proposing a "Spatial Aggregation Module" (SAM) to better integrate information across different scan phases at a pixel level, and an "Uncertain Region Inpainting Module" (URIM) to refine ambiguous tumor boundaries. The method demonstrates improved segmentation accuracy, particularly in challenging boundary regions.

What Problem Does This Paper Address?

Accurate segmentation of liver tumors from medical images is crucial for diagnosis and treatment planning. While multi-phase CT scans offer complementary information that can improve segmentation, current methods often struggle to effectively fuse this information. Existing approaches typically aggregate features across phases using simple weighting or channel-wise attention, neglecting the crucial spatial relationships between corresponding pixels in different phases. This can lead to suboptimal feature integration and redundancy. Furthermore, segmentation performance is often hampered by uncertainty, especially in tumor boundary regions, due to factors like lost high-frequency information during downsampling and low contrast between tumors and surrounding tissues. This paper aims to tackle these challenges by proposing a method that enhances cross-phase spatial feature aggregation and refines uncertain segmentation regions.

Methods

The proposed method builds upon a ResNeXt-50 backbone with two Siamese streams: one for the portal venous (PV) phase and one for the arterial (ART) phase. The core innovations are:

  • Spatial Aggregation Module (SAM): This module is designed to capture pixel-wise inter-phase relationships. It takes feature maps from both PV and ART streams and generates pixel-wise response maps. These maps are then used to modulate and fuse the features from each phase, ensuring that each pixel's representation benefits from its counterpart in the other phase. The SAM extracts descriptors using average and max pooling, then uses pyramid convolutional structures to model local and global inter-phase complementary relationships.
  • Uncertain Region Inpainting Module (URIM): This module focuses on refining ambiguous segmentation results, particularly at tumor boundaries. It leverages "confident" pixels (those with high segmentation scores) to "inpaint" neighboring uncertain pixels. This is achieved through a novel "local-confidence convolution" (LC-Conv) operation, which allows uncertain pixels to absorb discriminative features from surrounding confident pixels. The URIM module consists of multiple LC-Conv layers that progressively refine the features.

The overall network combines these modules with a standard encoder-decoder architecture, using cross-entropy losses for training.

Datasets

The authors evaluate their method on an in-house multi-phase CT dataset of focal liver lesions (MPCT-FLLs). This dataset comprises 121 multi-phase CT cases covering five typical liver tumor types: cysts, focal nodular hyperplasia (FNH), hemangiomas (HEM), hepatocellular carcinoma (HCC), and metastasis (METS). The dataset includes manual delineations for both liver and tumors. The authors employ a five-fold cross-validation technique for evaluation.

Key Insights and Ideas

  • Pixel-wise inter-phase relationships are critical: The paper highlights that simply concatenating or weighting features from different phases is insufficient. Explicitly modeling the spatial correspondence between pixels across phases, as done by the SAM, leads to more effective feature integration.
  • Uncertainty is a key bottleneck for boundary segmentation: The URIM module demonstrates that addressing segmentation uncertainty, especially in boundary regions, can significantly improve overall accuracy. The idea of using confident pixels to guide the refinement of uncertain ones is a powerful concept for image segmentation.
  • Modularity is beneficial: The SAM and URIM modules are designed to be pluggable, suggesting they can enhance other multi-phase segmentation architectures, not just the one presented here.

Key Results

The proposed method achieves state-of-the-art performance on the MPCT-FLLs dataset.

  • Overall performance: The method achieves the highest Dice per case (DPC) score of 80.12%, outperforming other methods like MC-FCN, MW-UNet, and PA-ResSeg. It also shows superior results across other metrics like Dice global (DG), VOE, RVD, ASD, and RMSD.
  • Tumor type performance: The method shows significant improvements across various tumor types, achieving the top average performance.
  • Ablation studies: The results from ablation studies confirm the effectiveness of each component. Adding multi-phase information boosts DPC by +6.46%, SAM improves it by +3.79%, and URIM contributes an additional +1.02%.
  • Robustness: When replacing the backbone with U-Net and ResNet-50, the SAM and URIM modules consistently improve segmentation performance, demonstrating their robustness.

Limitations and Future Work

  • Focus on two phases: The current method is designed for two-phase CT scans (PV and ART). The authors suggest that extending this to more than two phases would be a challenging but important future direction. This would likely require a more complex SAM module to generate response maps for each phase.
  • Memory consumption: Handling more phases could lead to increased memory consumption, which needs to be considered.
  • Generalizability: While tested on an in-house dataset, further validation on larger, diverse datasets would be beneficial.

Summary updated

Abstract

Multi-phase computed tomography (CT) images provide crucial complementary information for accurate liver tumor segmentation (LiTS). State-of-the-art multi-phase LiTS methods usually fused cross-phase features through phase-weighted summation or channel-attention based concatenation. However, these methods ignored the spatial (pixel-wise) relationships between different phases, hence leading to insufficient feature integration. In addition, the performance of existing methods remains subject to the uncertainty in segmentation, which is particularly acute in tumor boundary regions. In this work, we propose a novel LiTS method to adequately aggregate multi-phase information and refine uncertain region segmentation. To this end, we introduce a spatial aggregation module (SAM), which encourages per-pixel interactions between different phases, to make full use of cross-phase information. Moreover, we devise an uncertain region inpainting module (URIM) to refine uncertain pixels using neighboring discriminative features. Experiments on an in-house multi-phase CT dataset of focal liver lesions (MPCT-FLLs) demonstrate that our method achieves promising liver tumor segmentation and outperforms state-of-the-arts.
BibTeX
@inproceedings{zhang2021multi,
  author = {Yue Zhang and Chengtao Peng and Liying Peng and Huimin Huang and Ruofeng Tong and Lanfen Lin and Jingsong Li and Yen-Wei Chen and Qingqing Chen and Hongjie Hu and Zhiyi Peng},
  title = {{Multi-phase Liver Tumor Segmentation with Spatial Aggregation and Uncertain Region Inpainting}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2021},
  url = {https://openreview.net/forum?id=DzCmD71IM9},
}
Figure