Cascaded MultiTask 3-D Fully Convolutional Networks for Pancreas Segmentation

Jie Xue, Kelei He, Dong Nie, Ehsan Adeli, Zhenshan Shi, Seong-Whan Lee, Yuanjie Zheng, Xiyu Liu, Dengwang Li, Dinggang Shen

Introduction

Pancreas segmentation in Computed Tomography (CT) scans is a critical yet challenging task in medical image analysis. The difficulty stems from three primary factors: the organ's small volume—typically occupying less than 0.5% of an abdominal scan—its high inter-patient variability in shape and location, and low-contrast boundaries that often blend with adjacent structures like the stomach, duodenum, and intestines. While two-stage deep learning models have improved localization, they often struggle to capture the complex, thin, and "wiggly" geometry of the pancreas.

Methodology

This paper proposes a two-stage cascaded framework designed to enhance geometric understanding through multitask learning.

Cascaded Framework

  1. Stage 1 ($C_1$): Coarse Localization. A 3-D Fully Convolutional Network (FCN) processes downsampled CT images to identify a bounding box around the pancreas, effectively filtering out irrelevant abdominal noise.
  2. Stage 2 ($C_2$): Fine Multitask Segmentation. Within the localized region, a 3-D Dense-U-Net performs two simultaneous tasks: voxel-wise segmentation (predicting pancreas vs. background) and skeleton extraction (a regression task predicting the organ's internal skeleton).

Multitask Learning and Skeletonization

The core innovation lies in treating segmentation and skeletonization as complementary tasks. While segmentation focuses on boundary detection, skeletonization captures the organ's internal topology. This dual focus forces the model to develop a robust internal representation of the pancreas's global shape, which helps maintain accuracy when boundaries are blurry.

3D visualization of the segmented pancreas structure.

The architecture incorporates dense connections to facilitate feature flow and a "Feature Consistency Module" to fuse low-level and high-level features effectively. Finally, a 3-D Fully Connected Conditional Random Field (CRF) is applied as a post-processing step to smooth labels and eliminate isolated false-positive clusters.

Experimental Results

The model was evaluated on the NIH Pancreas Dataset (82 scans) and an in-house dataset from Fujian Medical University (FMU, 59 scans) containing complex pathological cases with benign or malignant cysts.

Performance Metrics

On the NIH benchmark, the proposed method achieved a state-of-the-art average Dice Similarity Coefficient (DSC) of 86.4%, surpassing previous models like RSTN (84.5%). The model also demonstrated significant robustness on the FMU dataset, handling morphological deformations caused by pathology more effectively than standard networks.

Pancreas segmentation results across multiple planes (axial, sagittal, coronal) for two different cases.

Efficiency and Ablation

The localization stage ($C_1$) is highly efficient, requiring approximately 4 seconds on an NVIDIA TITAN XP GPU. Ablation studies confirmed that both skeleton guidance and dense connections are essential for achieving peak accuracy; removing these components led to a measurable drop in performance.

Discussion and Conclusion

The integration of skeletonization as a geometric prior provides a significant advantage in segmenting organs with high shape variability. By providing a structural "reference," the multitask approach prevents the model from under-estimating the organ volume when boundaries are indistinct. Future work may extend this "segmentation + skeleton" paradigm to other tubular or thin structures, such as blood vessels or the gastrointestinal tract. While the model is robust, extremely large tumors that completely disrupt the underlying skeleton remain a challenge, suggesting further refinement for advanced pathological cases.

Summary updated

Abstract

Automatic pancreas segmentation is crucial to the diagnostic assessment of diabetes or pancreatic cancer. However, the relatively small size of the pancreas in the upper body, as well as large variations of its location and shape in retroperitoneum, make the segmentation task challenging. To alleviate these challenges, in this article, we propose a cascaded multitask 3-D fully convolution network (FCN) to automatically segment the pancreas. Our cascaded network is composed of two parts. The first part focuses on fast locating the region of the pancreas, and the second part uses a multitask FCN with dense connections to refine the segmentation map for fine voxel-wise segmentation. In particular, our multitask FCN with dense connections is implemented to simultaneously complete tasks of the voxel-wise segmentation and skeleton extraction from the pancreas. These two tasks are complementary, that is, the extracted skeleton provides rich information about the shape and size of the pancreas in retroperitoneum, which can boost the segmentation of pancreas. The multitask FCN is also designed to share the low- and mid-level features across the tasks. A feature consistency module is further introduced to enhance the connection and fusion of different levels of feature maps. Evaluations on two pancreas datasets demonstrate the robustness of our proposed method in correctly segmenting the pancreas in various settings. Our experimental results outperform both baseline and state-of-the-art methods. Moreover, the ablation study shows that our proposed parts/modules are critical for effective multitask learning.
BibTeX
@article{xue2021cascaded,
  author = {Jie Xue and Kelei He and Dong Nie and Ehsan Adeli and Zhenshan Shi and Seong-Whan Lee and Yuanjie Zheng and Xiyu Liu and Dengwang Li and Dinggang Shen},
  title = {{Cascaded MultiTask 3-D Fully Convolutional Networks for Pancreas Segmentation}},
  journal = {IEEE Trans. Cybernetics},
  year = {2021},
  doi = {10.1109/tcyb.2019.2955178},
}
Figure