Action Learning for 3D Point Cloud Based Organ Segmentation

Xia Zhong, Mario Amrehn, Nishant Ravikumar, Shuqing Chen, Norbert Strobel, Annette Birkhold, Markus Kowarschik, Rebecca Fahrig, Andreas MaierarXiv 2018 · arXiv

Here is a concise, engaging summary of the paper "Action Learning for 3D Point Cloud Based Organ Segmentation".

TL;DR

This paper introduces Active Point Net, a novel 3D organ segmentation pipeline that treats mesh deformation as a reinforcement learning problem. By predicting piece-wise linear transformations directly on point clouds using a new "aperture feature" extraction method, it achieves robust organ segmentation in under 3 seconds—making it up to 200 times faster than traditional dense voxel-labeling methods like U-Net, without requiring transfer learning for new scanners or protocols.

What Problem Does This Paper Address?

3D organ segmentation in CT scans is vital for computer-aided diagnosis, treatment planning, and radiation dose management. Currently, deep learning approaches are dominated by dense volumetric segmentation networks (like 3D U-Net). However, these methods suffer from three major drawbacks:

  1. Lack of Geometric Constraints: They label individual voxels, often requiring complex morphological post-processing to ensure the final shape looks like an actual organ.
  2. Dataset Sensitivity: They struggle to generalize across different CT scanners, imaging protocols, and contrast agents, requiring extensive fine-tuning or transfer learning.
  3. High Computational Cost: Segmenting a full 3D volume can take around 100 seconds on high-end GPUs, which is too slow for real-time interventional imaging.

The authors tackle these challenges by shifting the paradigm from voxel classification to mesh deformation via action learning.

Methods

The authors reformulate segmentation as a Reinforcement Learning (RL) problem using Deep Q-learning. Instead of minimizing a traditional energy function, an artificial agent learns to iteratively deform an initial 3D mesh until it fits the organ boundary.

Key components of the method include:

  • Aperture Features: To capture the image context around the mesh, the authors introduce a novel feature extraction technique. For each vertex, the volume is sampled within a "cone of vision" projected along the vertex's normal vector. This captures local grayscale intensities without processing the entire 3D volume.
  • Aperture Deformation Pyramid: The pipeline handles multi-resolution meshes hierarchically. It decomposes deformations into global affine transformations (translation, rotation, scaling) and local non-rigid transformations (moving vertices along their normals).
  • Network Architecture: Inspired by PointNet and U-Net, the architecture uses a shared multi-layer perceptron (MLP) to compute permutation-invariant local features, which are then aggregated with global features to predict the optimal deformation actions.
Fig. 1: The pipeline of our method (left) and illustration of aperture features (right). The training phase steps are blue, testing phase steps are red and joint steps are green. The aperture boundary is illustrated as a red cone, where the blue surface mesh is the current estimate.
Fig. 2: Illustration of the network architecture used to update a current object surface mesh estimate. Featuremaps are represented as boxes.

Datasets

The network was trained on 66 full-body CT volumes (with 200 data augmentations per volume via statistical shape models).

To evaluate robustness, it was tested across several distinct and challenging datasets:

  • Visceral dataset
  • MICCAI LiTS 2017 (Liver Tumor Segmentation) challenge data
  • SilverCopus (Thorax-Abdomen contrast-enhanced CTs)
  • DECT (Dual-Energy CT datasets)
  • Visible Human (VisHum) project (used as an unseen reference mesh evaluation)

Key Insights and Ideas

  • Learning Actions, Not Labels: By framing segmentation as a sequence of geometric actions (deformations), the network inherently preserves the topological properties of the organ shape.
  • Implicit Generalization: Because the agent learns how to move vertices based on local boundaries rather than memorizing global intensity distributions, it generalizes remarkably well. It successfully segments organs across completely different modalities (like flat-panel CT or dual-energy CT) without any fine-tuning.
  • Decoupling the Reference Mesh: The method can take an entirely unseen anatomical model (e.g., from the Visible Human project) and successfully deform it to fit a patient's unique 3D CT data.

Key Results

  • Speed: The runtime ranges from 0.3 to 2.7 seconds per organ on a modest NVIDIA K2100M GPU, representing a 20x to 200x speedup over dense volumetric methods.
  • Accuracy: Despite the massive speedup, accuracy remains highly competitive. The model achieved a Dice score of 89.3% to 89.5% for the liver on the LiTS and Anatomy3 datasets, and up to 94.3% for lung segmentation.
  • Robustness: The system successfully converged even when the initial mesh was randomly translated by up to 8 centimeters away from the true organ location.

Limitations and Future Work

While Active Point Net is exceptionally fast and robust across diverse scanning protocols, the authors note that it does not outright outperform state-of-the-art dense segmentation networks that have been trained and optimized specifically for a single dataset.

For future work, the authors suggest extending this method to generate full 3D dense deformation fields from the surface mesh using interpolation techniques like thin-plate splines. They believe this framework holds immense promise for real-time interventional imaging spaces where rapid feedback is critical.

Summary updated

Abstract

We propose a novel point cloud based 3D organ segmentation pipeline utilizing deep Q-learning. In order to preserve shape properties, the learning process is guided using a statistical shape model. The trained agent directly predicts piece-wise linear transformations for all vertices in each iteration. This mapping between the ideal transformation for an object outline estimation is learned based on image features. To this end, we introduce aperture features that extract gray values by sampling the 3D volume within the cone centered around the associated vertex and its normal vector. Our approach is also capable of estimating a hierarchical pyramid of non rigid deformations for multi-resolution meshes. In the application phase, we use a marginal approach to gradually estimate affine as well as non-rigid transformations. We performed extensive evaluations to highlight the robust performance of our approach on a variety of challenge data as well as clinical data. Additionally, our method has a run time ranging from 0.3 to 2.7 seconds to segment each organ. In addition, we show that the proposed method can be applied to different organs, X-ray based modalities, and scanning protocols without the need of transfer learning. As we learn actions, even unseen reference meshes can be processed as demonstrated in an example with the Visible Human. From this we conclude that our method is robust, and we believe that our method can be successfully applied to many more applications, in particular, in the interventional imaging space.
BibTeX
@misc{zhong2018action,
  author = {Xia Zhong and Mario Amrehn and Nishant Ravikumar and Shuqing Chen and Norbert Strobel and Annette Birkhold and Markus Kowarschik and Rebecca Fahrig and Andreas Maier},
  title = {{Action Learning for 3D Point Cloud Based Organ Segmentation}},
  year = {2018},
  eprint = {1806.05724},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1806.05724},
}
Figure