Consistent Cell Tracking in Multi-frames with Spatio-Temporal Context by Object-Level Warping Loss

Junya Hayashida, Kazuya Nishimura, Ryoma BiseWACV 2022 · OpenReview

Abstract

Multi-object tracking is essential in biomedical image analysis. Most methods follow a tracking-by-detection approach that involves using object detectors and learning the appearance feature models of the detected regions for association. Although these methods can learn the appearance similarity features to identify the same objects among frames, they have difficulties identifying the same cells because cells have a similar appearance and their shapes change as they migrate. In addition, cells often partially overlap for several frames. In this case, even an expert biologist would require knowledge of the spatial-temporal context in order to identify individual cells. To tackle such difficult situations, we propose a cell-tracking method that can effectively use the spatial-temporal context in multiple frames by using long-term motion estimation and an object-level warping loss. We conducted experiments showing that the proposed method outperformed state-of-the-art methods under various conditions on real biological images.

Introduction

Tracking cells in time-lapse microscopy is a cornerstone of biomedical research, yet it presents unique challenges compared to tracking macroscopic objects like vehicles. Cells are visually similar, lack distinct features, and undergo constant deformation. Furthermore, they frequently cluster or overlap, creating ambiguous boundaries that confuse standard detection algorithms. During mitosis, a single object splits into two, requiring the tracking system to handle topological changes seamlessly.

Difficult tracking scenarios where cells overlap require spatio-temporal context to maintain identity.

Standard "tracking-by-detection" methods often fail in these scenarios because they rely on local temporal context—typically only two adjacent frames. If a cell is missed or misidentified in a single frame due to overlap, the track is broken. This research proposes a method that leverages long-term spatio-temporal context to resolve such ambiguities by looking at many frames at once.

Methodology

The proposed framework utilizes a 3D Convolutional Neural Network (3D CNN) with a U-Net-like architecture to extract features from a volume of time-lapse images simultaneously. This allows the model to learn from a broader temporal window rather than isolated frame pairs.

Multi-Task Learning and Long-Term Motion

The network is trained to produce three distinct outputs for each frame: position heatmaps (indicating cell locations), short-term motion (displacement between $t$ and $t-1$), and long-term motion (displacement relative to a fixed reference frame $T$). By estimating motion over longer intervals, the system can "fill in the blanks" if a cell is temporarily obscured or missed in a single frame.

Object-Level Warping Loss

A key innovation is the Object-Level Warping Loss. Unlike traditional optical flow that warps raw pixels, this method warps the predicted position heatmaps. The estimated heatmap from one frame is transformed using the predicted motion vectors to align with the target frame. Any discrepancy between the warped heatmap and the target frame's estimated heatmap results in a penalty. This ensures temporal consistency and forces the network to learn physically plausible trajectories across the entire sequence.

The proposed framework uses a 3D CNN for long-term motion estimation and an object-level warping loss to ensure temporal consistency.

Handling Mitosis

The model naturally handles cell division by treating motion in the inverse direction (from child to parent). In this scheme, two daughter cells "point" back to the same mother cell, allowing the system to maintain lineage without complex association logic.

Experimental Results

The method was evaluated on a large-scale dataset of phase-contrast microscopy images featuring various chemical treatments (BMP2, FGF2, etc.) that alter cell density and behavior. The FGF2 condition was particularly challenging due to high cell density and frequent overlaps.

In comparative tests, the proposed method achieved a detection F1-score of 0.956 and a Target Effectiveness (TE) of 0.931, significantly outperforming state-of-the-art trackers like MPM and CenterTrack. Notably, in dense and difficult conditions, the method reduced tracking errors by 45% compared to previous benchmarks.

Conclusion

By integrating long-term motion estimation with a novel object-level warping loss, this method effectively utilizes spatio-temporal context to overcome the limitations of traditional cell tracking. While currently constrained by GPU memory to a 6-frame window, the approach demonstrates superior robustness in handling cell overlaps and dense clusters. Future work will focus on extending temporal consistency across even longer sequences as hardware capabilities evolve.

Summary updated

Abstract

Multi-object tracking is essential in biomedical image analysis. Most methods follow a tracking-by-detection approach that involves using object detectors and learning the appearance feature models of the detected regions for association. Although these methods can learn the appearance similarity features to identify the same objects among frames, they have difficulties identifying the same cells because cells have a similar appearance and their shapes change as they migrate. In addition, cells often partially overlap for several frames. In this case, even an expert biologist would require knowledge of the spatial-temporal context in order to identify individual cells. To tackle such difficult situations, we propose a cell-tracking method that can effectively use the spatial-temporal context in multiple frames by using long-term motion estimation and an object-level warping loss. We conducted experiments showing that the proposed method outperformed state-of-the-art methods under various conditions on real biological images.
BibTeX
@inproceedings{hayashida2022consistent,
  author = {Junya Hayashida and Kazuya Nishimura and Ryoma Bise},
  title = {{Consistent Cell Tracking in Multi-frames with Spatio-Temporal Context by Object-Level Warping Loss}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
  url = {https://openreview.net/forum?id=JcnEomzIdJ},
}
Figure