Abdominal multi-organ segmentation with organ-attention networks and statistical fusion

Yan Wang, Yuyin Zhou, Wei Shen, Seyoun Park, Elliot K. Fishman, Alan L. Yuille

TL;DR

This paper introduces a highly efficient framework for abdominal multi-organ CT segmentation by combining 2D Organ-Attention Networks with Reverse Connections (OAN-RC) across three orthogonal views. By leveraging a novel structural similarity-based statistical fusion method, the framework reconstructs precise 3D segmentations that outperform computationally heavy 3D deep learning models.

What Problem Does This Paper Address?

Segmenting multiple abdominal organs from CT scans is a critical step for computer-aided diagnosis and surgery. However, doing this automatically is notoriously difficult due to three main factors:

  1. Weak Boundaries: Spatially adjacent organs (like the pancreas and the duodenum) often share poorly defined borders with low soft-tissue contrast.
  2. Extreme Scale Variations: Organs vary drastically in size—compare the massive liver to the tiny gallbladder. Standard deep networks often fail on smaller structures because lower convolutional layers lack semantic context.
  3. The 3D Complexity vs. Memory Tradeoff: While CT scans are 3D volumes, training full 3D deep networks is severely limited by GPU memory and a scarcity of dense 3D expert annotations. Patch-based 3D approaches exist but often lose global context and introduce boundary discontinuities.

The authors tackle these challenges by asking: Can we achieve the accuracy of 3D contextual processing using efficient, holistic 2D networks?

Methods

The authors propose a multi-planar framework that breaks down the 3D segmentation task into 2D chunks and intelligently fuses them back together.

Figure 1. The overall framework.

The pipeline consists of two primary components:

1. Two-Stage Organ-Attention Network with Reverse Connections (OAN-RC)

Instead of segmenting the entire complex abdomen in one shot, the network uses a two-stage approach for each 2D slice:

  • Stage-I generates a preliminary organ probability map. This map is transformed into an Organ-Attention Map (OAM), which highlights target organ regions and suppresses complex background noise.
  • Stage-II takes a combination of the original image and the OAM, forcing the network to focus only on discriminating organs from their local backgrounds.
  • Reverse Connections (RC): To prevent small organs from getting lost, the authors introduce reverse connections in Stage-I. These connections pass high-level semantic information from deeper layers back down to lower, fine-scale layers, enriching them with the context needed to detect smaller structures.
Figure 2. The architecture of our two-stage organ-attention network with reverse connections. The organ-attention network (OAN) is composed of two jointly optimized stages, where the first stage (stage-I) transforms the organ segmentation probability map by spatial attention to the second stage (stage-II). Hence the organ segmentation map generated in the organ-attention module guides the latter computation. The reverse connections, described in Sec. 2.2, modify the first stage of OAN as shown by dashed lines.

2. Local Structural Similarity-Based Statistical Fusion (LSSF)

The OAN-RC is run independently across three orthogonal views: axial ($Z$), sagittal ($X$), and coronal ($Y$). To resolve non-identical predictions and enforce 3D connectivity, the authors use an Expectation-Maximization (EM) statistical fusion framework. Crucially, they weight the fusion using a Structural Similarity (SSIM) index that measures how well the local 2D slice properties correspond to the original 3D volume structure.

Datasets

  • FELIX Project Dataset: The framework was trained and evaluated on 236 normal abdominal CT scans collected for pancreatic cancer research.
  • Annotations: A total of 13 structures (including the aorta, colon, duodenum, gallbladder, IVC, kidneys, liver, pancreas, small bowel, spleen, stomach, and large veins) were manually annotated by four human raters and verified by a senior expert.
  • Evaluation: The approach was validated using a 4-fold cross-validation strategy.

Key Insights and Ideas

  • Attention Simplifies the Background: By utilizing a two-stage attention mechanism, the network transforms a complex multi-class global problem into a series of simpler local discrimination tasks.
  • Bridging the Scale Gap: Reverse connections act as a feature-refinement mechanism. By feeding abstract semantic concepts back to early layers, the network naturally adapts to multi-scale organ structures without requiring separate, scale-specific models.
  • 2D Efficiency + 3D Fusion > Pure 3D: Standard 3D networks are bottlenecked by local patch processing due to memory limits. Processing full 2D slices captures global context within the slice plane, while direction-dependent statistical fusion successfully recovers the lost cross-sectional 3D connectivity.

Key Results

The proposed OAN-RC with Local Similarity-based Statistical Fusion (LSSF) consistently outperformed state-of-the-art 3D patch-based methods (like 3D U-Net and Hierarchical 3D FCN) and standard 2D majority-voting methods.

  • Dice-Sørensen Similarity Coefficient (DSC) Improvements:
  • Pancreas: Reached 87.8% accuracy compared to 79.3% for 3D U-Net.
  • Aorta: Achieved 91.8% accuracy.
  • Liver & Kidneys: Maintained exceptionally high accuracies (98.0% and 96.8% / 98.4% respectively).
  • Surface Distance Reduction: The LSSF method significantly minimized boundary noise, yielding much smoother and anatomically realistic 3D surfaces.
  • Inference Speed: The entire 13-organ segmentation pipeline takes less than 1 minute per volume (averaging ~76.73 seconds for multi-view testing and just ~6.87 seconds for the 3D statistical fusion step), making it highly practical for clinical workflows.

Limitations and Future Work

  • Prior Probability Dependencies: The fusion step relies on a calculated spatial prior $P(T)$. If the underlying deep networks suffer from systematic over- or under-estimation biases, the final fusion quality could degrade.
  • Dataset Homogeneity: The dataset was collected with controlled scanning parameters from a single institution. The authors note that performance might vary when encountering significant artifacts or different reconstruction kernels from diverse clinical settings.
  • Future Directions: Exploring alternative prior models to handle systematic network biases could further robustify the statistical fusion step.

Summary updated

Abstract

Accurate and robust segmentation of abdominal organs on CT is essential for many clinical applications such as computer-aided diagnosis and computer-aided surgery. But this task is challenging due to the weak boundaries of organs, the complexity of the background, and the variable sizes of different organs. To address these challenges, we introduce a novel framework for multi-organ segmentation by using organ-attention networks with reverse connections (OAN-RCs) which are applied to 2D views, of the 3D CT volume, and output estimates which are combined by statistical fusion exploiting structural similarity. OAN is a two-stage deep convolutional network, where deep network features from the first stage are combined with the original image, in a second stage, to reduce the complex background and enhance the discriminative information for the target organs. RCs are added to the first stage to give the lower layers semantic information thereby enabling them to adapt to the sizes of different organs. Our networks are trained on 2D views enabling us to use holistic information and allowing efficient computation. To compensate for the limited cross-sectional information of the original 3D volumetric CT, multi-sectional images are reconstructed from the three different 2D view directions. Then we combine the segmentation results from the different views using statistical fusion, with a novel term relating the structural similarity of the 2D views to the original 3D structure. To train the network and evaluate results, 13 structures were manually annotated by four human raters and confirmed by a senior expert on 236 normal cases. We tested our algorithm and computed Dice-Sorensen similarity coefficients and surface distances for evaluating our estimates of the 13 structures. Our experiments show that the proposed approach outperforms 2D- and 3D-patch based state-of-the-art methods.
BibTeX
@article{wang2019abdominal,
  author = {Yan Wang and Yuyin Zhou and Wei Shen and Seyoun Park and Elliot K. Fishman and Alan L. Yuille},
  title = {{Abdominal multi-organ segmentation with organ-attention networks and statistical fusion}},
  journal = {Medical Image Analysis},
  year = {2019},
  doi = {10.1016/j.media.2019.04.005},
}
Figure