Letting Time Settle Annotator Disagreement in Cell Tracking

When several experts label the same microscopy image, they often disagree. One sees two touching cells where another sees one. One marks a faint blob as a dying cell and another calls it debris. In a single still frame these calls are often undecidable. In a time-lapse video, many of them are not: two sister cells drift apart a few frames later, and debris does not move like a cell or divide. This proposal treats each annotator's version of each frame as a competing hypothesis. A global tracking optimizer then picks, jointly across all frames, the set of hypotheses that forms the most coherent cell trajectories. A global tracking optimizer here means a solver that links detections across the whole video at once rather than frame by frame.

The same machinery serves a second purpose. At inference time, the raw overlapping proposals a detector produces before it prunes duplicates enter the same graph as extra hypotheses. The tracker then replaces non-maximum suppression (NMS), the standard greedy step that deletes any box overlapping a higher-scoring box. Greedy NMS is known to wrongly delete real cells in dense clusters. The output is threefold: cleaner training labels than majority voting produces, a tracker that never commits to a single pruned detection map, and a short queue of the few disagreements that temporal evidence cannot resolve, handed to a human inside a viewer biologists already use.

Core Claim

The claim is that temporal trajectory consistency adjudicates inter-annotator disagreement better than static consensus methods. "Static consensus" here covers majority vote, per-pixel fusion, and filtering annotators by their agreement with an anchor expert. The specific prediction concerns contested candidates, meaning objects or boundaries that annotators label differently in a frame. On these, a global min-cost-flow solver over the union of all annotators' instances will pick the version matching held-out gold truth more often than majority vote does. Detectors trained on these adjudicated labels will also track better downstream.

The closest existing work is Large-Scale Multi-hypotheses Cell Tracking Using Ultrametric Contours Maps. It already jointly selects segments and links them over time with an integer linear program, so "replace NMS with a global multi-hypothesis tracker" is not new by itself. That method differs from this proposal in three ways. Its hypotheses come from a nested watershed hierarchy, so conflicts are simple ancestor–descendant relations. Its costs are pure geometric overlap, which the authors themselves flag as a place where learned weights are unexplored. And it never ingests human annotations. Here, hypotheses come from non-nested sources: several humans plus a detector's raw proposals. Non-nested sources need general overlap-conflict constraints. Their costs come from measured rater support. The scientific object of study is the label-adjudication question, not only tracking accuracy.

Research Program

The graph is built per time window. Every instance drawn by any annotator, plus every pre-NMS detector proposal, becomes a candidate node $v$ with an estimated support $p_v$. Support combines the fraction of annotators who drew a matching instance with the detector's calibrated score. Edges connect candidates in adjacent frames, or skip a few frames, scored by position, shape and appearance in the tunable style of LapTrack: linear assignment particle tracking with tunable metrics. Any group of candidates that cover the same pixels forms a conflict set $K$, inside which at most one candidate may be active. Writing $x_v \in \{0,1\}$ for whether candidate $v$ is kept and $y_e \in \{0,1\}$ for whether link $e$ is used, the solver minimizes

$$ \sum_v -\log\frac{p_v}{1-p_v}\, x_v \;+\; \sum_e c_e\, y_e \quad \text{s.t.}\quad \sum_{v\in K} x_v \le 1 \;\; \forall K, $$

where $c_e$ is the link cost. Flow conservation is also imposed: a kept node has one incoming link, or is a track start, and one or two outgoing links, where two links means a division. Without the conflict constraints this is the network-flow formulation of Globally-optimal greedy algorithms for tracking a variable number of objects. The conflict sets make it an integer program, which we solve exactly on sliding windows and by linear relaxation with rounding for long videos. We reuse the windowing and spatial-indexing tricks the ultrametric paper used to reach millions of instances.

The work has three parts. First, the solver and a learned version of $p_v$ and $c_e$. These are fit on gold-annotated frames, closing the learned-weights gap that the ultrametric paper leaves open. Second, a multi-annotator time-lapse benchmark, which the file shows does not currently exist. Every multi-rater study it covers is static, including Variability Matters: Evaluating Inter-Rater Variability in Histopathology for Robust Cell Detection and Supporting Mitosis Detection AI Training with Inter-Observer Eye-Gaze Consistencies. We will collect five independent annotators on dense subsequences of the Cell Tracking Challenge datasets and of the CTMC dataset.

As a second, cheaper source of disagreement, we treat the individual algorithm outputs that The Cell Tracking Challenge: 10 years of objective benchmarking fused into its silver standard as separate "raters". That benchmark raised coverage from 17.8% to 99.1%, but it flattened its sources into one mask, and we keep the sources separate. Third, a napari plugin built on LapTrack's ability to freeze verified links. We extend freezing to candidate choice and rank remaining disagreements by how much the optimal cost changes if the decision flips. Only disagreements whose flip barely changes that cost go to the user.

Each prior line of work plays a distinct role. CTMC motivates the problem by showing detection is the bottleneck. Falcon Perception-HD: High Density Perception via Reinforcement Learning and The Four Color Theorem for Cell Instance Segmentation remove NMS within a single image, and serve as single-frame comparison points rather than methods we reuse. Variability Matters constrains the design: its finding that conformity-filtered subsets beat the full discordant set warns that naive pooling of raters can hurt, so the support term $p_v$ must be calibrated per annotator, not a raw vote fraction.

Baselines

The naive reference fuses annotators by per-pixel majority vote. It trains a detector on those labels, prunes with standard NMS at a tuned threshold, and links with LapTrack's two-stage assignment. This is what a lab would do today. For context, CTMC reports Tracktor with Faster R-CNN detections at MOTA 23.59, against 99.32 for DeepSORT on ground-truth detections (CTMC: Cell Tracking with Mitosis Detection Dataset Challenge). That gap is the room any detection-side improvement can occupy.

The strongest existing method is the ultrametric multi-hypothesis tracker of Large-Scale Multi-hypotheses Cell Tracking Using Ultrametric Contours Maps. It ranked first on 3 of 5 3D Cell Tracking Challenge datasets (CE, TRIF, DRO); the file gives a ranking but no scalar score. It is the strongest competitor because it already escapes single-map NMS by choosing among hypotheses across time. We run it on majority-vote labels and on the same detector backbone's outputs, with the same compute and the same hyperparameter search budget, using LapTrack's Optuna-style tuning for both systems.

For association quality we also compare against TRACKASTRA: Transformer-Based Cell Tracking for Live-Cell Microscopy, which reports error counts (AOGM) of 23 on BACTERIA versus 118 for Delta 2.0, and 7.9 on DynamicNuclearNet. For label fusion we compare against conformity filtering in the style of Variability Matters. That paper shows filtered subsets outperform the full set, but no numeric detection score appears in the file.

Beating the strongest baseline means two things on the Cell Tracking Challenge sequences we annotate. First, a statistically significant gain in tracking accuracy (TRA) and detection accuracy (DET) over the ultrametric tracker with the same inputs. Second, higher agreement with held-out gold on contested candidates than majority vote.

Three ablations isolate the key ingredient. The first removes temporal links, solving each frame alone. This turns the method into a static adjudicator and tests whether time is doing the work. The second removes annotator candidates and keeps only detector proposals, which tests whether multi-rater input adds anything beyond a better NMS replacement. The third replaces learned support with raw vote fractions.

Why not just…?

Why not just take a majority vote and then track with the best global solver? This is the real threat, and it is the comparison the whole proposal hinges on. Majority vote throws away minority hypotheses before the tracker sees them. If the correct reading is a minority view, for example only two of five annotators split a touching pair that later separates, no downstream solver can recover it. CTMC and Trackastra both note that a missed detection cannot be tracked. The per-frame ablation and the majority-then-ultrametric baseline together settle this. If temporal adjudication does not beat majority vote on contested candidates, the claim is false and we say so.

Why not just filter for high-conformity annotators, as Variability Matters does? Conformity filtering works on static patches. It rewards agreement with an anchor annotator chosen by annotation volume, which the file notes may entrench that anchor's systematic errors. It discards whole annotators rather than deciding case by case. Our support term can absorb per-annotator reliability, so we include conformity-weighted support as a variant. If it matches the full method, the temporal term is shown to be unnecessary.

Why not just use Soft-NMS, an adaptive threshold, or a single-image NMS-free detector like Falcon Perception-HD or four-color segmentation? These fix pruning within one frame, and Falcon Perception-HD's reported gain of +21.5 F1 points in scenes with more than 300 objects shows that matters. None of them can use the fact that two overlapping blobs move apart later, and none ingests multiple annotators. We run a four-color segmentation model through the official FCIS codebase as an alternative detector feeding LapTrack. If it closes most of the gap, the contribution shrinks to label adjudication alone.

Proposed Experiments

The decisive first experiment is cheap. It uses the separate algorithm outputs behind the Cell Tracking Challenge silver standard as raters and the human gold frames as truth. On every candidate where the sources disagree, we measure how often the full solver, the per-frame ablation, and majority vote match gold. We expect the full solver to win specifically on split/merge and appear/disappear disagreements, and to tie on pure boundary-placement disputes, where time carries little information. If it does not win on the first category, we stop.

If it passes, we collect the five-annotator subsequences and repeat the test with human disagreement. We then train the same detector backbone on majority-vote, conformity-filtered, and temporally adjudicated labels. We evaluate SEG, TRA and DET on held-out Cell Tracking Challenge sequences, and MOTA and IDF1 on CTMC. The inference-time variant, with pre-NMS proposals inside the graph, is evaluated against NMS plus LapTrack and against the ultrametric tracker. Finally, a small user study in the napari plugin measures how many decisions per video a biologist must review, and how long reaching gold-level TRA takes. The comparison condition is correcting LapTrack output by hand. CTMC's human tracking accuracy of about 0.95 serves as the ceiling.