Tracking cells in label-free microscopy looks like a linking problem, but it is almost entirely a detection problem. On DIC (differential interference contrast) time-lapse video, where cell edges fade smoothly into the background and optical shear creates halos, a standard tracker reaches 99.32 MOTA (multi-object tracking accuracy) when given perfect detections. It falls to 23.59 MOTA when given the output of an ordinary Faster R-CNN detector (CTMC). Each new cell line or microscope also needs its own detector, and annotations are exactly what is missing. We propose to run that dependency in reverse. A hierarchical tracker first joins confident detections into short fragments, then links those fragments across gaps. That global view knows things no single frame does. It can tell where a cell must have been during a frame the detector missed, and which confident-looking blobs never persist and are therefore debris or halos. We turn those judgments into training labels and use them to adapt the detector to the new domain, starting from zero to five annotated frames.
In plain terms, the detector proposes, the tracker checks against time, and the detector is retrained on what the tracker certified. What is new is the source of the labels. They come from long-range, gap-closing association, not from per-frame confidence. They also include the detector's own misses and its own confident false alarms. Those are precisely the two errors that low-contrast boundaries cause, and frame-level self-training cannot see either.
Core Claim
Hierarchical tracklet association produces three kinds of supervision that are more reliable than the detector's own confidence in a low-contrast target domain. The first is gap-filled positives: positions interpolated between two linked tracklets across frames where the detector fired weakly or not at all. The second is orphan negatives: confident detections that belong to no tracklet surviving global linking. The third is temporally calibrated boundary softness: how much a cell's box extent varies along its tracklet, used to set how sharply its boundary is supervised.
We claim that adapting a detector on these labels, with few trainable parameters, closes a substantial part of the gap between automated and oracle detections on held-out DIC cell lines. We further claim that it beats confidence-thresholded self-training and few-shot fine-tuning run under the same label budget. The claim is falsifiable at a specific point. If the gap-filled positives are not more precise than the detector's own low-confidence proposals at the same locations, the idea fails.
Relation to Prior Work
The motivating evidence comes from benchmarks. CTMC benchmarks 86 DIC videos over 14 cell lines and isolates detection as the bottleneck. It also shows deep trackers generalizing poorly across cell lines: DeepCell reaches TRA (tracking accuracy) 0.32, below classical Viterbi at 0.39, against roughly 0.95 for humans. It proposes no adaptation method. The Cell Tracking Challenge reports the same "generalizability wall" across modalities. It extends mask coverage from 17.8% to 99.1% by fusing dozens of competing algorithms into "silver truth," a resource that does not exist in a new domain. Our proposal replaces the ensemble of algorithms with an ensemble over time within one algorithm.
The association machinery is reused, not invented. Cell Population Tracking and Lineage Construction with Spatiotemporal Context builds conservative local tracklets, then retrospectively links them with bipartite matching, recovering a cell hidden for 10 frames. Addressing Ambiguity in Multi-target Tracking by Hierarchical Strategy and GMMCP establish that linking fragments, rather than raw detections, keeps global optimization tractable. UnOVOST is the closest structural precedent. It lowers detector thresholds, builds optical-flow tracklets, and prunes them with a tree search over tracklets. However, it uses tracking only to produce output and never feeds the decisions back into the COCO-trained detector, which degrades off RGB. Feeding back is the step we add.
Two lines of work constrain the design. TRACKASTRA links cells over 6-frame windows from coordinates alone and admits it cannot close long gaps. That motivates using a hierarchical linker as the teacher instead of a short-window one. It also supports its finding that motion outweighs appearance when cells change shape. CTMC's DeepSORT failure on noisy DIC appearance points the same way, so our gap-closing step relies mainly on motion. Segment Anything for Cell Tracking argues the opposite, linking with SAM2 appearance memory zero-shot, though greedily and with weaker biological accuracy. We treat this tension as an experimental question rather than taking a side. ADVMEM is a negative result we take seriously. Test-time adaptation on temporally correlated tracklet streams collapses; ROTTA's error rises from 27.6% to 66.8%. We therefore adapt in offline rounds over shuffled frames from all target videos, not online along the stream.
The adaptation and boundary components are borrowed and kept small. CAT-SAM and CoPEFT show that tuning a small set of parameters (adapters or prompt tokens) adapts large models from one image or 10% of target data. CaPro adapts from a single unlabeled image. It filters low-contrast false alarms with an MNIST-trained shape codebook, a prior that suits curvilinear structures and not round cells. Our orphan-negative rule plays the same filtering role, but it uses temporal persistence instead of a hand-chosen shape prior. Bounding Maps for Universal Lesion Detection replaces hard boxes with soft maps that fall from 1.0 at the center to 0.5 at the edge. It gains 3.85 sensitivity points on DeepLesion. We adopt soft targets, but set each target's falloff from that tracklet's temporal extent variance instead of a fixed ramp.
Research Program
The system has three parts, run in alternating rounds. First, a detector is pretrained on source cell lines and then run on unlabeled target video at a deliberately low threshold, so that faint cells are proposed along with clutter. Second, a two-tier tracker processes these proposals. The lower tier links only high-overlap, motion-consistent detections into short tracklets and leaves crossings and divisions unlinked. The upper tier links tracklets with an assignment solver that uses motion (and, in an ablation, appearance) costs and allows division events, since CTMC annotates mitosis.
Third, the upper tier's decisions become labels. When two tracklets are linked across a gap of $g$ frames, the interpolated positions in those frames become positive boxes. Confident detections whose tracklet stays shorter than a length $L$ after global linking, and which no link absorbs, become negatives. Detections inside long tracklets become positives with soft boundary targets. Each box edge is supervised with a ramp whose width scales with the standard deviation of that edge's position, after motion compensation, along the tracklet. Stable cells thus get sharp targets and cells that blur into the background get soft ones.
Only a small adapter set in the detector is trained on these labels, plus the $k \in \{0, 1, 5\}$ annotated target frames when available. We use two or three rounds, with gap length and margin thresholds tightened toward conservative values. The analysis centers on the precision and recall of each label type against held-out ground truth as a function of gap length $g$. That curve tells us where the tracker is a trustworthy teacher and where it starts propagating its own errors.
Baselines
The naive reference point is the source-trained detector applied unchanged to the held-out cell line, followed by the same two-tier tracker. This measures the domain gap directly. For context, CTMC reports Tracktor with Faster R-CNN at MOTA 23.59, DeepCell at TRA 0.32 and Viterbi at TRA 0.39. Those are CTMC's own protocol, not our leave-one-line-out split, so we rerun rather than copy them. The oracle-detection tracker, at MOTA 99.32 with IDF1 94.23 in CTMC, is the ceiling. We report the fraction of the naive-to-oracle gap each method closes.
The strongest existing competitor is few-shot parameter-efficient fine-tuning of the same detector on the $k$ labeled target frames. We use CAT-SAM-style conditional adapters (CAT-SAM), which report 85.2% average one-shot mIoU against 79.3% for LoRA across 11 segmentation datasets. There is no reported result on Lacuna for any few-shot adapted detector on CTMC or DIC tracking. It is strongest because it is the established way to use scarce target labels and needs no video. It runs with the same backbone, the same adapter parameter count, the same labeled frames, and an equal tuning budget measured in GPU hours and hyperparameter trials. We also include LoRA as the simpler variant.
The second competitor is zero-shot SAM2 cell tracking (Segment Anything for Cell Tracking), which reports TRA 0.921 on the fluorescence dataset Fluo-N3DL-TRIF. There is no reported result on Lacuna for it on CTMC or DIC. It is prompted from the same source detector's boxes.
Two ablations isolate the key ingredient. The first is confidence self-training: the same adapter, rounds and budget, with pseudo-labels taken from detections above a confidence threshold and no tracking. The second is short-window teaching: labels from the lower tier only, or from a Trackastra-style 6-frame linker, with no gap closure. Beating them means higher target MOTA, IDF1 and TRA than every same-budget competitor at each $k$, with the margin over short-window teaching coming specifically from gap-filled positives, measured through detection recall in frames the naive detector missed.
Why not just…?
Why not just fine-tune the detector on the few labeled frames? This is the main threat when $k=5$, because DIC cells within one line look alike and five frames contain hundreds of cells. The evidence against relying on it alone is indirect. CAT-SAM needs an annotated image and still requires prompts, and CTMC's cross-line failures suggest that per-frame learning misses line-specific clutter. The direct test is our $k$ sweep. If few-shot fine-tuning matches our method at $k=5$, the contribution shrinks to the $k \in \{0, 1\}$ regime, and we will report that plainly. We also combine the two, since they are complementary rather than exclusive.
Why not just self-train on the detector's confident outputs? This is the ablation that could kill the idea. In low contrast, the detector's confident errors are halos and debris, and its misses are faint cells. Confidence thresholding keeps the first and never recovers the second, and this is the false-positive failure the file documents for boundary branches. Temporal persistence is a different signal. The decisive measurement is precision of pseudo-labels at matched recall. If confidence labels are as precise as tracker labels, the tracker adds nothing.
Why not just track with SAM2 zero-shot? CaPro and CAT-SAM report vanilla SAM collapsing on low-contrast domains: 15.60% mIoU on OCT1600 and 78.5% on JSRT. SAM2 tracking links greedily, which hurts long-horizon consistency. We run it anyway on CTMC because it is not documented to have been run there, and a strong result would be informative in its own right.
Why not just use a short-window linker as the teacher? Short windows yield cleaner but fewer labels, and they cannot label a cell missed for several frames. The gap-length curve settles this: gains should concentrate at $g$ beyond the window size.
Proposed Experiments
The main benchmark is CTMC under leave-one-cell-line-out. We adapt to each of the 14 lines from the other 13, with $k \in \{0, 1, 5\}$ labeled target frames and the line's unlabeled videos, reporting MOTA, IDF1 and TRA. A secondary transfer study uses the Cell Tracking Challenge DIC and phase-contrast datasets, with DET and TRA, to test the cross-modality wall.
The decisive checks fall in the order of the argument. First, gap-filled positives must beat the naive detector's low-confidence proposals in precision at the same locations. Second, orphan negatives must be mostly clutter when checked against ground truth. Third, the method must beat confidence self-training at equal budget. Fourth, a motion-only versus appearance-augmented gap-closing comparison bears directly on the Trackastra versus SAM2 tension. Fifth, across rounds, we check whether identity errors compound; ADVMEM's collapse is the failure pattern we are guarding against. The soft boundary targets are tested last, against fixed Bounding-Map ramps and hard boxes, because they are the least essential part of the proposal.
References
- CTMC: Cell Tracking with Mitosis Detection Dataset Challenge
- The Cell Tracking Challenge: 10 years of objective benchmarking
- Cell Population Tracking and Lineage Construction with Spatiotemporal Context
- Addressing ambiguity in multi-target tracking by hierarchical strategy
- GMMCP tracker: Globally optimal Generalized Maximum Multi Clique problem for multiple object tracking
- UnOVOST: Unsupervised Offline Video Object Segmentation and Tracking
- TRACKASTRA: Transformer-Based Cell Tracking for Live-Cell Microscopy
- Segment Anything for Cell Tracking
- ADVMEM: Adversarial Memory Initialization for Realistic Test-Time Adaptation via Tracklet-Based Benchmarking
- CAT-SAM: Conditional Tuning for Few-Shot Adaptation of Segment Anything Model
- CoPEFT: Fast Adaptation Framework for Multi-Agent Collaborative Perception with Parameter-Efficient Fine-Tuning
- CaPro: Curvilinear-aware Prompt Learning with Single Unlabeled Image for Cost-effective Curvilinear Structure Segmentation
- Bounding Maps for Universal Lesion Detection