Star-convex Polyhedra for 3D Object Detection and Segmentation in Microscopy

Martin Weigert, Uwe Schmidt, Robert Haase, Ko Sugawara, Gene MyersWACV 2020

TL;DR

This paper introduces STARDIST-3D, a novel method for accurately detecting and segmenting cell nuclei in challenging 3D microscopy images. By representing nuclei as star-convex polyhedra and employing an efficient deep learning approach, STARDIST-3D significantly outperforms existing methods, particularly in datasets with low signal-to-noise ratios and dense cell packing.

What Problem Does This Paper Address?

Accurate segmentation of cell nuclei in 3D microscopy is crucial for many biological studies, from understanding developmental processes to tracing cell lineages. However, existing automated methods often struggle with common challenges in microscopy data: low signal-to-noise ratios (making it hard to distinguish nuclei from background) and dense packing of nuclei (making it difficult to separate individual cells). While deep learning has shown promise, methods that rely on simple bounding boxes or pixel-wise semantic segmentation can still fail to accurately delineate individual, often irregularly shaped, nuclei in these complex scenarios.

Methods

The core innovation of STARDIST-3D is the use of star-convex polyhedra to represent cell nuclei. Unlike simpler shapes like spheres or bounding boxes, star-convex polyhedra offer a more flexible and accurate way to describe the often irregular shapes of biological objects.

The method builds upon a previous 2D approach (STARDIST) by extending it to 3D. Key methodological contributions include:

  • Efficient Shape Representation: The authors devised a way to represent 3D star-convex polyhedra using a reduced number of radial directions (64, down from a naive 1024) while still achieving faithful shape reconstruction. This was achieved by using rays based on a spherical Fibonacci lattice and adapting them to account for anisotropic voxel sizes common in microscopy.
  • Deep Learning Backbone: A modified 3D ResNet architecture is used to densely predict, for each pixel in a downsampled grid, the object probability and the distances to the object boundary along the chosen radial directions.
  • Efficient Non-Maximum Suppression (NMS): A critical challenge in 3D is efficiently pruning the redundant predicted shapes. STARDIST-3D employs a novel NMS strategy that uses successively tighter bounds (bounding spheres, inscribed spheres, convex hulls, kernels) to quickly discard overlapping candidates, resorting to exact intersection computation only when necessary. This makes the method practical for large 3D volumes.

Datasets

The authors evaluated STARDIST-3D on two distinct datasets:

  • WORM: A dataset of DAPI-stained nuclei from C. elegans first larval stage, featuring near-isotropic resolution and semi-automatically annotated instances.
  • PARHYALE: A dataset of nuclei from Parhyale hawaiensis expressing Histone-EGFP. This dataset is characterized by highly anisotropic resolution (especially along the Z-axis), a lower signal-to-noise ratio, and fewer labeled training images, making it more challenging.

Key Insights and Ideas

  • Star-convexity is powerful for biological shapes: The paper demonstrates that representing cell nuclei with star-convex polyhedra is a more robust approach than simpler geometric primitives, especially for complex and irregular shapes.
  • Anisotropy matters: Accounting for anisotropic voxel sizes in microscopy data is crucial for accurate shape representation and segmentation. The method's ability to adapt its shape representation based on automatically computed anisotropy factors significantly improves performance.
  • Efficient NMS is key for 3D: The proposed bounding-based NMS significantly reduces computational cost, making 3D instance segmentation feasible for large datasets.
  • Data efficiency: STARDIST-3D shows remarkable performance even with limited training data, outperforming other methods that require much more annotated data.

Key Results

STARDIST-3D consistently outperformed classical (IFT-Watershed) and deep learning (U-Net, U-Net+) baselines on both datasets.

  • Accuracy: For the challenging PARHYALE dataset, STARDIST-3D achieved significantly higher accuracy across various Intersection over Union (IoU) thresholds, particularly at higher thresholds. For example, at an IoU threshold of 0.5, STARDIST-3D achieved an accuracy of 0.593, compared to 0.247 for U-Net+ and 0.161 for IFT-Watershed.
  • Data Efficiency: When trained with only 4.15% of the data from the WORM dataset, STARDIST-3D achieved an accuracy of 0.7 at IoU=0.5, matching the performance of U-Net+ trained on 100% of the data.
  • Speed: The efficient NMS implementation makes STARDIST-3D practical for large 3D volumes, with a reported runtime of 9 seconds for a stack of size 1141x140x140 with 12000 initial candidates.

Limitations and Future Work

  • Detection Failures: While STARDIST-3D produces superior segmentations, it can sometimes fail to detect certain nuclei, especially in the PARHYALE dataset.
  • Generalizability: The paper suggests that STARDIST-3D should be generally applicable to segment objects whose shapes are well-represented by star-convex polyhedra, implying potential for broader applications beyond cell nuclei. Future work could explore this generalizability.
  • Further Optimization: While efficient, further optimizations in the NMS or network architecture could potentially lead to even faster inference times or improved accuracy.

Summary updated

BibTeX
@inproceedings{weigert2020star,
  author = {Martin Weigert and Uwe Schmidt and Robert Haase and Ko Sugawara and Gene Myers},
  title = {{Star-convex Polyhedra for 3D Object Detection and Segmentation in Microscopy}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
}
Figure