SwinCell: a transformer-based framework for dense 3D cellular segmentation

Xiao Zhang, Zihan Lin, Liguo Wang, Yong S. Chu, Yang Yang, Xianghui Xiao, Yuewei Lin, Qun LiubioRxiv 2024 · DOI

TL;DR

This paper introduces SwinCell, a novel 3D segmentation framework that leverages the Swin Transformer architecture to achieve highly accurate segmentation of dense cellular structures in 3D images. By effectively capturing both local and global contextual information, SwinCell significantly outperforms existing methods, paving the way for more robust and automated cellular analysis in biology and tissue engineering.

What Problem Does This Paper Address?

Accurately segmenting cells in 3D is crucial for understanding biological processes, but it's a notoriously difficult task. This is especially true for dense tissues where cells are tightly packed, leading to complex image features, varying cell shapes, and overlapping boundaries. Existing 2D-based methods, when extended to 3D by stitching slices, often introduce artifacts and struggle with maintaining consistency across different planes. The challenge lies in developing a model that can effectively interpret the intricate 3D spatial context and distinguish individual cell instances within these complex environments.

Methods

SwinCell builds upon the strengths of both the Swin Transformer and the Cellpose algorithm. The core innovation is the integration of the Swin Transformer architecture into an encoder-decoder framework (similar to U-Net). This allows SwinCell to leverage the Swin Transformer's ability to capture long-range contextual information through its hierarchical structure and shifted window self-attention mechanism.

The framework also incorporates the "flow prediction" approach from Cellpose. This involves predicting gradient flows that indicate the direction and magnitude of pixel displacement towards cell centers. These predicted flows, along with a cell probability map, are then used with a gradient tracking method to delineate individual cell instances. The model is trained end-to-end, predicting four channels simultaneously: three flow maps (XYZ) and one cell probability map.

Datasets

SwinCell was evaluated on a combination of public and in-house datasets:

  • Colon tissue dataset: A public dataset containing synthetic 3D digital phantoms of human colon tissue, characterized by densely connected cells.
  • Allencell (hiPS) dataset: A public dataset of live-cell images, where the DNA cell nucleus channel was used for evaluation.
  • In-house 3D cellular holotomography dataset: Data acquired using a Nanolive 3D microscope, featuring HEK293 adherent cells with complex internal structures.

Key Insights and Ideas

  • Swin Transformer for 3D Context: The paper demonstrates that the Swin Transformer's ability to capture both local and global context is a significant advantage for 3D cellular segmentation, outperforming traditional CNNs with limited receptive fields.
  • Flow Prediction Synergy: Combining the Swin Transformer's contextual understanding with the Cellpose flow prediction mechanism leads to more accurate and consistent flow predictions, which are critical for robust instance segmentation.
  • End-to-End 3D Processing: SwinCell's unified 3D approach avoids the artifacts and inconsistencies often introduced by slice-stitching methods used in 2D-to-3D extensions.
  • Generalizability: The framework shows strong performance across diverse datasets, including synthetic phantoms, nuclei, and complex cellular structures, highlighting its potential as a general-purpose 3D segmentation tool.

Key Results

SwinCell consistently outperformed the state-of-the-art Cellpose algorithm across various datasets and metrics:

  • Allen Dataset: SwinCell achieved a recall of 0.83, precision of 0.97, and F1 score of 0.81, significantly outperforming Cellpose variants which produced more false positives and struggled with certain planes.
  • Colon Dataset: SwinCell accurately segmented 427 out of 459 cells with 32 false positives (precision 0.93, recall 0.93). In contrast, Cellpose produced 306 true positives and 751 false positives (precision 0.29, recall 0.67). SwinCell's predicted cell volumes and diameters were not significantly different from ground truth, unlike Cellpose.
  • Nanolive Dataset: SwinCell achieved an averaged precision of 0.84, recall of 0.88, and F1 score of 0.86, while Cellpose achieved lower scores (precision 0.30, recall 0.81, F1 0.43), particularly struggling with XZ and YZ planes.
  • Ablation Study: Comparing SwinCell with a U-Net model (Extended Data Fig. 4) showed that SwinCell predicts smoother and more consistent flows, leading to higher accuracy and fewer false positives.

Limitations and Future Work

  • Labeled Data Scarcity: The authors acknowledge that the availability of labeled 3D datasets remains a significant challenge for developing universally applicable "generalist" models. They addressed this by using synthetic data generation and human-in-the-loop refinement.
  • Computational Constraints: While Swin-transformer reduces computational complexity compared to traditional transformers, 3D models still require substantial computational resources.
  • Future Directions: The authors suggest self-supervised learning as a promising avenue for future work, leveraging the Swin-transformer's flexibility to pre-train models on unlabeled data for improved performance and broader applicability. They aim to develop a comprehensive, universally adaptable 3D segmentation model.

Summary updated

Abstract

Abstract Segmentation of three-dimensional (3D) cellular images is fundamental for studying and understanding cell structure and function. However, 3D cellular segmentation is challenging, particularly for dense cells and tissues. This challenge arises mainly from the complex contextual information within 3D images, anisotropic properties, and the sensitivity to internal cellular structures, which often lead to miss-segmentation. In this work, we introduce SwinCell, a 3D transformer-based framework that leverages Swin-transformer for flow prediction and effectively distinguishes individual cell instances in 3D. We demonstrate the broad utility of the SwinCell in the segmentation of nuclei, colon tissue cells, and dense cultured cells. SwinCell strikes a balance between maintaining detailed local feature recognition and understanding broader contextual information. Tested extensively with both public and in-house 3D cell imaging datasets, SwinCell shows superior performance in segmenting dense cells in 3D, making it a powerful 3D segmentation tool for cellular analysis that could expedite research in cell biology and tissue engineering.
BibTeX
@misc{zhang2024swincell,
  author = {Xiao Zhang and Zihan Lin and Liguo Wang and Yong S. Chu and Yang Yang and Xianghui Xiao and Yuewei Lin and Qun Liu},
  title = {{SwinCell: a transformer-based framework for dense 3D cellular segmentation}},
  year = {2024},
  doi = {10.1101/2024.04.05.588365},
  howpublished = {bioRxiv},
}
Figure