CellSeg3D: self-supervised 3D cell segmentation for fluorescence microscopy

Cyril Achard, Timokleia Kousi, M. M. Frey, Maxime Vidal, Yves Paychère, Colin Hofmann, Asim Iqbal, Sebastien B Hausmann, Stéphane Pagès, Mackenzie Weygandt Mathis

TL;DR

This paper introduces CellSeg3D, a Python package and napari plugin for 3D cell segmentation in fluorescence microscopy. It features a novel self-supervised learning method (WNet3D) that achieves performance comparable to state-of-the-art supervised methods, even without ground truth labels. This is particularly impactful given the scarcity of labeled 3D microscopy data, and the authors also release a new, fully annotated mesoSPIM dataset to facilitate future research.

What Problem Does This Paper Address?

Accurately segmenting and quantifying cells in 3D fluorescence microscopy images is crucial for understanding biological processes, especially in fields like neuroscience. However, generating high-quality 3D ground truth data for training segmentation models is extremely time-consuming and labor-intensive. This scarcity of labeled data limits the development and application of advanced segmentation techniques, particularly for large-scale 3D datasets like those from light-sheet microscopy (LSM). Existing methods often rely on 2D segmentation followed by 3D reconstruction, which can lead to information loss and errors. Direct 3D segmentation is desirable but hampered by the lack of sufficient 3D annotated datasets.

Methods

The core of this work lies in two main contributions: a new self-supervised learning method called WNet3D and a comprehensive software package, CellSeg3D.

  • WNet3D: This is a novel 3D self-supervised learning model for semantic segmentation. It's inspired by the W-Net architecture but adapted for 3D volumes. Key modifications include:
  • A full conversion to a 3D architecture.
  • The use of a 3D SoftNCuts loss for segmentation and a reconstruction loss (MSE or BCE) for the decoder.
  • A weighted sum of encoder and decoder losses, updated in a single backward pass, for improved stability.
  • A reduced network depth (three up/downsampling steps) and the use of group normalization instead of batch normalization for better performance and speed.
  • Crucially, WNet3D is trained without any human-annotated ground truth labels, learning from the inherent structural properties of the unlabeled 3D volumes.
  • CellSeg3D: This is a Python package and napari GUI plugin that provides access to WNet3D and other state-of-the-art supervised models (SwinUNetR, SegResNet, Cellpose, StarDist). It offers a complete workflow for:
  • Data annotation (including a 3D annotator).
  • Training (both supervised and self-supervised).
  • Model inference.
  • Post-processing and evaluation.
  • The package aims to make advanced 3D cell segmentation accessible to researchers.

Datasets

The authors utilized several datasets for training and benchmarking:

  • Newly developed mesoSPIM dataset: This is a key contribution. It's a fully human-annotated 3D dataset of mouse brain volumes acquired using a mesoSPIM system, containing over 2,600 TPH2-positive cells. This dataset is intended to serve as a benchmark for 3D cell segmentation.
  • Platynereis-ISH-Nuclei confocal dataset: An existing dataset used for benchmarking.
  • 3D Platynereis-Nuclei light-sheet dataset: Another existing dataset for evaluation.
  • Mouse-Skull-Nuclei confocal dataset: A challenging dataset with densely packed nuclei, also used for benchmarking.

Key Insights and Ideas

  • Self-supervised learning can match supervised performance: The most striking insight is that WNet3D, trained entirely without human labels, achieves performance on par with or even exceeding state-of-the-art supervised methods on challenging 3D datasets. This significantly lowers the barrier to entry for 3D cell segmentation.
  • The scarcity of 3D ground truth is a major bottleneck: The paper strongly emphasizes that the lack of large, high-quality 3D annotated datasets is a primary limitation in the field, and their new dataset aims to address this.
  • Direct 3D segmentation is superior: The authors advocate for direct 3D segmentation over 2D-to-3D reconstruction methods, as it preserves more morphological information and can streamline processing.
  • Active learning for semi-supervised approaches: The paper demonstrates how self-supervised models like WNet3D can be used to generate "pseudo-labels" that can then be corrected and used to train supervised models, offering a powerful active learning strategy for label-scarce scenarios.
  • Generalizability of self-supervised models: WNet3D shows promising generalization capabilities across different datasets and imaging modalities, even when trained on a single dataset.

Key Results

  • WNet3D performance: On the newly developed mesoSPIM dataset, WNet3D (trained without labels) achieved an F1-score of 0.81±0.004 with artifact filtering and 0.74±0.12 without. This is comparable to supervised models trained with 80% of the data. In low-data regimes (10% training data), WNet3D significantly outperformed most supervised baselines, achieving an F1-score of 0.74±0.02.
  • Benchmarking: WNet3D demonstrated strong performance across all four tested datasets, often outperforming supervised methods, especially on the challenging Mouse-Skull-Nuclei dataset.
  • Software availability: The CellSeg3D package, including WNet3D, supervised models, and annotation tools, is publicly available, facilitating broader adoption.
  • New dataset: The release of the fully human-annotated mesoSPIM dataset provides a valuable resource for the community.

Limitations and Future Work

  • Artifacts: Self-supervised models like WNet3D cannot inherently distinguish between cells and artifacts. Post-processing filtering or active learning with human correction is necessary to address this.
  • Uncluttered nuclei focus: The current WNet3D is optimized for relatively uncluttered nuclei. While it performs well on challenging datasets like Mouse-Skull, further work might be needed for more complex scenarios.
  • Electron microscopy and axon tracing: The authors note that WNet3D has not been tested on electron microscopy data or for axon tracing, suggesting other tools might be more suitable for those specific tasks.
  • More datasets needed: While the authors provide a new dataset, they emphasize the need for more diverse 3D datasets to further advance the field.
  • Future work: The authors suggest adapting more complex methods for instance segmentation if cells are more overlapping and encourage the community to contribute additional annotations.

Summary updated

Abstract

Abstract Understanding the complex three-dimensional structure of cells is crucial across many disciplines in biology and especially in neuroscience. Here, we introduce a set of models including a 3D transformer (SwinUNetR) and a novel 3D self-supervised learning method (WNet3D) designed to address the inherent complexity of generating 3D ground truth data and quantifying nuclei in 3D volumes. We developed a Python package called CellSeg3D that provides access to these models in Jupyter Notebooks and in a napari GUI plugin. Recognizing the scarcity of high-quality 3D ground truth data, we created a fully human-annotated mesoSPIM dataset to advance evaluation and benchmarking in the field. To assess model performance, we benchmarked our approach across four diverse datasets: the newly developed mesoSPIM dataset, a 3D platynereis-ISH-Nuclei confocal dataset, a separate 3D platynereis-Nuclei light-sheet dataset, and a challenging and densely packed Mouse-Skull-Nuclei confocal dataset. We demonstrate that our self-supervised model, WNet3D – trained without any ground truth labels – achieve performance on par with state-of-the-art supervised methods, paving the way for broader applications in label-scarce biological contexts.
BibTeX
@inproceedings{achard2025cellseg,
  author = {Cyril Achard and Timokleia Kousi and M. M. Frey and Maxime Vidal and Yves Paychère and Colin Hofmann and Asim Iqbal and Sebastien B Hausmann and Stéphane Pagès and Mackenzie Weygandt Mathis},
  title = {{CellSeg3D: self-supervised 3D cell segmentation for fluorescence microscopy}},
  booktitle = {eLife},
  year = {2025},
  doi = {10.7554/elife.99848.4},
}
Figure