CMTT-JTracker: a fully test-time adaptive framework serving automated cell lineage construction

Liuyin Chen, Sanyuan Fu, Zijun ZhangBriefings in Bioinformatics 2024 · DOI

TL;DR

This paper introduces CMTT-JTracker, a novel framework for automated cell lineage construction that excels at adapting to diverse cell imaging conditions without requiring retraining. By combining a test-time adaptive segmentation module (CMTT) with a joint detection and ReID network, it achieves state-of-the-art performance in cell tracking while being computationally efficient and robust across various datasets.

What Problem Does This Paper Address?

Automated cell tracking is crucial for understanding cellular behavior in biology and medicine. However, existing methods face significant challenges:

  • Data Scarcity: Training robust cell tracking models requires large, well-annotated datasets, which are expensive and time-consuming to create.
  • Data Variability: Cell images vary greatly due to differences in imaging modalities (e.g., fluorescence, phase contrast), illumination, contrast, cell types, and densities. Models trained on one dataset often perform poorly on others.
  • Computational Cost: Many advanced methods are computationally intensive, making them impractical for biologists with limited resources.
  • Complex Scenarios: Dense cell populations, overlapping cells, and cell-cell contact make accurate segmentation and tracking difficult.

The authors aim to develop a cell tracking method that is accurate, computationally efficient, and generalizes well across diverse cell imaging datasets, even with limited or no target domain annotations.

Methods

The core of the proposed method is the CMTT-JTracker, a framework built upon the Joint Detection and Embedding (JDE) concept, enhanced with a novel fully test-time adaptive segmentation module (CMTT).

  • CMTT (Central-Metric Fully Test-Time Adaptive Framework): This module is designed for pre-segmentation of cell images. It operates in two stages during testing:
  • Stage 1 (Entire Image Adaptation): The entire target image is used to adapt a pre-trained segmentation model (U-Net++ in this case) using a combination of Shannon entropy and batch nuclear-norm loss. This stage helps align distributions and reduce uncertainty.
  • Stage 2 (Patch Adaptation): The target image is split into patches, and a central-metric contrastive (CMC) loss is applied. This loss leverages information learned in Stage 1 to better distinguish cells from the background and handle local image variations, especially in dense scenarios.
  • JDE-based Tracking Network: The feature maps from CMTT are fed into a multi-task learning network that simultaneously performs object detection and ReID.
  • Detection Branch: Standard convolutional layers produce detection results, followed by non-maximum suppression.
  • ReID Branch: A Spatial Attention ReID network (SA-ReID) is employed to aggregate features from different scales and enhance the learning of object-related representations, crucial for distinguishing individual cells.

The key innovation is the test-time adaptation of the segmentation module, allowing it to adapt to new, unseen data without requiring any further training or annotations from the target domain.

Datasets

The authors utilized several publicly available datasets for evaluation and introduced a new dataset:

  • CTC Datasets: Nine 2D datasets (e.g., Fluo-N2DH-SIM+, PhC-C2DL-PSC) were used for benchmarking tracking performance. These datasets vary in cell source, imaging density, and microscopy modality.
  • BBBC Datasets: A collection of microscopy image sets used for segmentation benchmarks.
  • EVICAN: A large-scale cell dataset with partially annotated images.
  • SWARM Dataset (Newly Introduced): A novel, densely annotated cell image dataset featuring large image sizes and densely packed cells, designed to challenge segmentation models. The authors note that this is the first dataset of its kind.

Key Insights and Ideas

  • Test-Time Adaptation for Segmentation is Key: The CMTT module's ability to adapt to unseen data during testing without retraining is a significant advantage, especially for diverse biological imaging.
  • Two-Stage Adaptation for Robustness: The combination of entire-image and patch-based adaptation in CMTT effectively handles domain shifts and complex scenarios like dense cell populations.
  • CMC Loss for Dense Cells: The central-metric contrastive loss is particularly effective in Stage 2 of CMTT for distinguishing cells in crowded environments.
  • SA-ReID Enhances Feature Learning: The spatial attention mechanism in the ReID branch helps consolidate multi-scale features, improving the ability to identify individual cells.
  • JDE with Adaptive Segmentation Outperforms: Integrating CMTT as the segmentation unit within JDE-based trackers significantly boosts performance compared to using standard segmentation methods like U-Net.
  • Biological Performance Matters: The authors emphasize that metrics like Complete Tracks (CT) and Track Fraction (TF) are crucial for evaluating the biological relevance of cell lineage construction, not just raw accuracy.

Key Results

  • State-of-the-Art Tracking Performance: CMTT-JTracker achieved the highest MOTA (Multiple Object Tracking Accuracy) on the Fluo-N2DH-SIM+ (0.894) and PhC-C2DL-PSC (0.850) datasets among the tested methods.
  • Improved Biological Metrics: The framework also showed superior performance in CT and TF metrics, indicating better reconstruction of complete cell lineages.
  • CMTT as a Superior Segmentation Unit: When CMTT was used as the segmentation module in other tracking-by-detection methods (e.g., KTH Tracker, MPM Tracker), it consistently improved their performance.
  • CMTT Segmentation Performance: As a standalone segmentation unit, CMTT outperformed state-of-the-art benchmarks on various datasets, especially in dense cell scenarios, achieving Dice coefficients as high as 0.928.
  • Robustness and Generalizability: The framework demonstrated strong performance across cross-cell-type, cross-microscopy-modality, and adaptation to small-scale and dense cell datasets.
  • Computational Efficiency: The test-time adaptation for CMTT takes only about 5 minutes for adaptation (less than 10 epochs) on an NVIDIA GPU 3080Ti, making it practical for users.

Limitations and Future Work

  • Data Limitations: While extensive experiments were conducted, the authors acknowledge that data from an even wider range of sources would further validate the conclusions. The limited number of annotated images in their new SWARM dataset, despite its large total size, might also affect representativeness.
  • Online Association Algorithm: The current work focuses on the segmentation and detection aspects. The online association algorithm for cell tracking could be further improved to comprehensively consider factors like cell division and cell death, which are not fully accounted for.
  • Future Work: The authors suggest that further research could explore more sophisticated online association algorithms and investigate the framework's applicability to 3D cell tracking. The proposed CMTT paradigm could also be extended to other computer vision tasks requiring domain adaptation.

Summary updated

BibTeX
@article{chen2024cmtt,
  author = {Liuyin Chen and Sanyuan Fu and Zijun Zhang},
  title = {{CMTT-JTracker: a fully test-time adaptive framework serving automated cell lineage construction}},
  journal = {Briefings in Bioinformatics},
  year = {2024},
  doi = {10.1093/bib/bbae591},
}
Figure