Author response: CellSeg3D: self-supervised 3D cell segmentation for fluorescence microscopy

Cyril Achard, Timokleia Kousi, M. M. Frey, Maxime Vidal, Yves Paychère, Colin Hofmann, Asim Iqbal, Sebastien B Hausmann, Stéphane Pagès, Mackenzie Weygandt MathisOpenalex Unknown 2025 · DOI · OpenAlex

Abstract

Understanding the complex three-dimensional structure of cells is crucial across many disciplines in biology and especially in neuroscience. Here, we introduce a set of models including a 3D transformer (SwinUNetR) and a novel 3D self-supervised learning method (WNet3D) designed to address the inherent complexity of generating 3D ground truth data and quantifying nuclei in 3D volumes. We developed a Python package called CellSeg3D that provides access to these models in Jupyter Notebooks and in a napari GUI plugin. Recognizing the scarcity of high-quality 3D ground truth data, we created a fully human-annotated mesoSPIM dataset to advance evaluation and benchmarking in the field. To assess model performance, we benchmarked our approach across four diverse datasets: the newly developed mesoSPIM dataset, a 3D platynereis-ISH-Nuclei confocal dataset, a separate 3D platynereis-Nuclei light-sheet dataset, and a challenging and densely packed Mouse-Skull-Nuclei confocal dataset. We demonstrate that our self-supervised model, WNet3D – trained without any ground truth labels – achieve performance on par with state-of-the-art supervised methods, paving the way for broader applications in label-scarce biological contexts.

Introduction: The 3D Annotation Bottleneck

In modern neuroscience and biology, imaging techniques like light-sheet microscopy (LSM) can produce massive 3D volumes of intact tissue, such as a whole mouse brain. However, the utility of this data is often limited by the "segmentation bottleneck"—the process of identifying and outlining individual cells within these vast volumes.

The current gold standard for segmentation relies on supervised deep learning, which requires thousands of human-annotated examples. In 3D, this task is exceptionally labor-intensive; labeling a single volume can take weeks of expert time. While some researchers attempt to segment 2D slices and stack them, this approach often fails to capture complex 3D morphological information. CellSeg3D addresses this challenge by introducing a framework that learns to recognize cells directly from raw data without requiring manual labels.

Methodology: WNet3D and Self-Supervision

The core innovation of this work is WNet3D, a self-supervised autoencoder architecture designed for volumetric data. Unlike supervised models that learn from human-provided "answers," WNet3D learns by attempting to reconstruct the input data.

The Architecture

WNet3D consists of two 3D U-Nets joined end-to-end. The first U-Net (the encoder) predicts a semantic segmentation mask. The second U-Net (the decoder) then attempts to reconstruct the original image using only that predicted mask. To ensure the model finds meaningful biological structures rather than noise, the researchers utilized a SoftNCuts loss function. This mathematical objective encourages the model to group voxels (3D pixels) that are similar in brightness and spatial proximity while separating them from the background.

Technical Refinements

To adapt the original 2D WNet concept for 3D microscopy, several adjustments were made:

  • Group Normalization: Replaced standard Batch Normalization to better handle the small batch sizes necessitated by memory-heavy 3D data.
  • Optimized Depth: The model depth was trimmed to ensure fast inference times without sacrificing accuracy.
  • User Accessibility: The entire pipeline is integrated into CellSeg3D, a Python package and napari plugin that allows researchers to perform training, inference, and manual refinement through a graphical user interface (GUI).

Benchmarking and Datasets

To validate the self-supervised approach, the study utilized four diverse datasets, including a new high-quality benchmark:

  1. MesoSPIM Dataset: A high-resolution 3D dataset of cleared mouse brain tissue containing 2,632 manually annotated neurons (TPH2-tdTomato). This served as the primary benchmark for comparing self-supervised vs. supervised methods.
  2. Platynereis Nuclei: Confocal and light-sheet datasets used to test the model's generalizability across different imaging modalities.
  3. Mouse Skull Nuclei: A challenging, densely packed dataset used to test the limits of the segmentation logic.
  4. c-FOS Dataset: Whole-brain mouse samples stained for c-FOS to evaluate performance on noisier, real-world biological signals.

Key Results

Parity with Supervised Learning

The most significant finding is that WNet3D achieves a Dice coefficient of approximately 0.81 on the mesoSPIM dataset. This performance is on par with state-of-the-art supervised models like Cellpose and StarDist, even when those models are specifically tuned for the data.

Robustness in Low-Data Scenarios

Supervised models typically fail when training data is scarce. In tests where supervised models were restricted to only 10% of available training data, WNet3D outperformed them by nearly 2x in semantic segmentation accuracy. Because WNet3D learns from the entire raw volume, it is far more robust for niche biological samples where no pre-existing ground truth exists.

Self-Supervised Distillation

The authors demonstrated a hybrid workflow: using WNet3D to generate "silver standard" labels for a raw dataset, and then using those labels to train a supervised model like SwinUNetR (a 3D transformer). This approach yields results nearly as accurate as those achieved with human-annotated training data, effectively automating the creation of training sets.

Limitations and Future Directions

While WNet3D represents a major step forward, some challenges remain:

  • Intensity Dependence: Because the model relies on brightness to group voxels, it can occasionally struggle to distinguish actual cells from bright artifacts or "halos" without additional post-processing.
  • Hyperparameter Sensitivity: Optimal results currently require manual tuning of intensity and spatial sigma values for specific tissue types.
  • Crowded Environments: In extremely dense samples, such as those found in electron microscopy, intensity-based self-supervision still faces difficulties in "de-clumping" adjacent cells.

Future work will focus on automating hyperparameter selection and improving the model's ability to resolve individual instances in high-density tissues. By providing an open-source tool through the napari ecosystem, CellSeg3D offers a scalable solution for the growing demands of volumetric microscopy.

Summary updated

Abstract

Understanding the complex three-dimensional structure of cells is crucial across many disciplines in biology and especially in neuroscience. Here, we introduce a set of models including a 3D transformer (SwinUNetR) and a novel 3D self-supervised learning method (WNet3D) designed to address the inherent complexity of generating 3D ground truth data and quantifying nuclei in 3D volumes. We developed a Python package called CellSeg3D that provides access to these models in Jupyter Notebooks and in a napari GUI plugin. Recognizing the scarcity of high-quality 3D ground truth data, we created a fully human-annotated mesoSPIM dataset to advance evaluation and benchmarking in the field. To assess model performance, we benchmarked our approach across four diverse datasets: the newly developed mesoSPIM dataset, a 3D platynereis-ISH-Nuclei confocal dataset, a separate 3D platynereis-Nuclei light-sheet dataset, and a challenging and densely packed Mouse-Skull-Nuclei confocal dataset. We demonstrate that our self-supervised model, WNet3D – trained without any ground truth labels – achieve performance on par with state-of-the-art supervised methods, paving the way for broader applications in label-scarce biological contexts.
BibTeX
@inproceedings{achard2025author,
  author = {Cyril Achard and Timokleia Kousi and M. M. Frey and Maxime Vidal and Yves Paychère and Colin Hofmann and Asim Iqbal and Sebastien B Hausmann and Stéphane Pagès and Mackenzie Weygandt Mathis},
  title = {{Author response: CellSeg3D: self-supervised 3D cell segmentation for fluorescence microscopy}},
  booktitle = {Openalex Unknown},
  year = {2025},
  doi = {10.7554/elife.99848.3.sa0},
}
Figure