Which Pseudo-Label Errors Matter for Mask-Based Trackers

Some video trackers outline each object with a pixel mask instead of a rectangle. The mask lets the tracker learn the object's appearance from object pixels only, so background in the corners of a box does not leak into the model. These masks are expensive to annotate by hand, so the strongest recent systems train on masks generated automatically by other networks and optical flow, the so-called "silver-standard" labels. Such data can beat hand-labelled data on a single aggregate score. Nobody has measured what kinds of errors these generated masks contain, or which of those errors actually damage tracking.

We propose to make dataset transparency for automatically mined tracking data quantitative. We would publish a small set of measured error rates, each split by error type, that are shown to predict how a mask-based tracker trained on the data will fail. The core hypothesis is that error type matters more than overall mask accuracy. Masks that bleed into the background at their edges should hurt a tracker's ability to keep identities apart, and this harm should be largest in trackers that build identity features from mask pixels alone. The same overall mask overlap, made up instead of missing occluded segments or mis-linked tracks, should hurt different parts of performance.

Core Claim

Learning Multi-Object Tracking and Segmentation from Automatic Annotations does three things. It builds KITTI Synth, roughly 44,000 images and 1.25 million segments, with a pipeline of instance segmentation, structure-from-motion-trained optical flow and mask warping. It introduces MOTSNet, whose "mask-pooling" layer pools appearance features only inside the predicted mask. It reports that training on this silver data beats training on the hand-cleaned KITTI MOTS labels, by +1.9% sMOTSA on cars and +7.5% on pedestrians.

That paper motivates this proposal. It shows silver data helps, but it does not characterize the label noise. The authors also note that the pipeline breaks under long occlusions.

Reproducibility of deep learning in digital pathology whole slide image analysis supplies the methodological lens. It reimplemented three Camelyon pipelines from their papers alone and found that undocumented preprocessing caused large collapses, for example lesion detection FROC falling from 0.981 to 0.499. Its response was a 12-point reporting checklist. A checklist records what was done; it does not measure what the data contains.

Our claim is that, for mined mask datasets, the useful transparency artifact is an "error profile": measured rates of boundary leakage, missed or occluded segments, and identity-linking errors, estimated against a gold subset. The profile is useful only if, together with a sensitivity curve per error type, it predicts which tracking sub-scores will drop. If error types with matched overall overlap produce indistinguishable downstream damage, the claim is false, and a single mask-IoU number would be enough.

Research Program

The first step creates aligned silver and gold labels on identical frames. We rerun the published three-phase mining pipeline on the KITTI MOTS training sequences, which carry hand-corrected masks from MOTS: Multi-Object Tracking and Segmentation. We then match every mined segment to its gold counterpart.

Each discrepancy is assigned to one of four classes. Boundary errors are the pixels gained or lost at the mask edge, measured with boundary IoU, an overlap score computed only in a thin band around the contour. Missing segments are false negatives, stratified by occlusion level. Spurious segments are false positives. Linking errors are identity switches and fragmentations in the mined tracklets. The error profile is the vector of these rates, reported separately per class (cars and pedestrians) and per occlusion bin.

The second step is controlled noise injection. Starting from gold KITTI MOTS labels, we corrupt each error type separately at several rates. Boundary leakage is simulated by dilating masks into background along flow-consistent directions. Segment loss is targeted at occluded instances. Linking errors are random tracklet merges and splits. The rates are calibrated so that conditions are matched on total mask-level IoU against the clean labels. This matching is what separates "error type" from "error amount."

Each corrupted dataset trains two trackers with identical backbones and budgets. The first is MOTSNet with mask-pooling. The second is the same network with standard rectangular region pooling. Comparing the two isolates the spatial-masking ingredient. We predict that mask-pooling makes identity features depend directly on mask edges in the training labels, so boundary noise should hurt it more than the box-pooled model.

We evaluate with HOTA from HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking, computed with mask IoU. HOTA splits performance into localization (LocA), detection (DetA) and association (AssA). It is used here because, unlike sMOTSA's single composite $(\tilde{TP}-FP-IDS)/M$, it can show whether a given label error damages identity keeping or pixel placement. That paper found MOTA correlates 0.96 with detection but only 0.46 with association, so a single score would hide exactly the effect we are testing.

The third step tests prediction and intervention. We fit simple per-type sensitivity curves of each sub-metric against error rate. We then feed in the measured profile of the real mined labels and check whether predicted losses match the observed gap between silver-trained and gold-trained models on the same frames. Finally, we filter the mined data, removing segments flagged for the error type the curves identify as most harmful. Filtering uses an automatic flag (flow-warp inconsistency and mask-edge disagreement with an independent segmenter), so it can be applied to the full KITTI Synth set without gold labels. We compare the result against removing the same number of random segments.

Baselines

The naive reference point is current practice. Train MOTSNet on the full mined dataset and report sMOTSA plus a qualitative description of the pipeline, with no error profile at all. Everything we build has to say something this reference cannot.

The gold-only reference is TrackR-CNN trained on KITTI MOTS, as reported in MOTS: Multi-Object Tracking and Segmentation: validation sMOTSA 76.2 on cars and 46.8 on pedestrians. We retrain it under our tuning budget rather than trusting cross-paper comparability.

The strongest existing method is MOTSNet trained on KITTI Synth, from Learning Multi-Object Tracking and Segmentation from Automatic Annotations. It reports +1.9% sMOTSA over TrackR-CNN on cars, +7.5% on pedestrians and +4.1% on MOTSChallenge; the file gives no absolute MOTSNet scores. We reproduce it with the released settings. All models share data, schedule length, hyperparameter search budget and five random seeds, and differences are reported with seed-level confidence intervals.

Beating this baseline means two things. A profile-guided filtered training set should improve HOTA or AssA over full KITTI Synth training by more than seed variance. It should also beat random filtering at the same data reduction.

The key ablation is mask-pooling against rectangular region pooling under each noise type. It tests whether spatial reliability masking specifically amplifies boundary-label noise.

A second ablation evaluates the same models with sMOTSA only. It shows whether the composite score would have hidden the type-specific effects.

Why not just…?

Why not just report the mean IoU of the pseudo-labels against a gold subset? This is the real competitor, and the matched-IoU injection experiment exists to settle it. Suppose boundary leakage, segment loss and linking errors at equal total IoU produce statistically indistinguishable HOTA sub-scores across seeds. Then the single number suffices, and the proposal shrinks to recommending that number. We expect otherwise. A mask that leaks into the background should corrupt identity features in a mask-pooled model, while a missing occluded segment mainly removes detection supervision. But we treat this as the hypothesis under test, not a given.

Why not just use Fell et al.'s reproducibility checklist? The checklist records preprocessing choices and split details. That would capture KITTI Synth's pipeline parameters but not its error content. The intervention experiment measures the difference: a checklist alone gives no guidance on which segments to remove, while the profile plus sensitivity curves does.

Why not just train on more silver data, or mix in gold, and let scale absorb the noise? Porzi et al.'s result is consistent with scale winning on average. If scale had absorbed all type-specific harm, the silver-trained model would show no sub-metric gap relative to gold on matched frames, and filtering would not beat random removal. Both outcomes are measured directly. A gold-plus-silver mix is also trained as an extra control at equal compute.

Why not just use HOTA on masks and skip the dataset work? We do use HOTA unchanged; the metric is not our contribution. HOTA diagnoses the trained model but cannot say which property of the training data caused a given AssA loss. Linking that loss to a measured data property is what this proposal adds.

Proposed Experiments and Decisive Checks

The benchmarks are KITTI MOTS (10,870 frames, 977 objects) for aligned silver and gold profiling and noise injection, and MOTSChallenge for testing on crowded pedestrian scenes, where boundary leakage between adjacent people should matter most.

Metrics are mask-IoU HOTA with LocA, DetA and AssA, sMOTSA for comparability with prior reports, and boundary IoU of predicted masks. No reported result was found on Lacuna for HOTA or boundary IoU on KITTI MOTS for either TrackR-CNN or MOTSNet, so we compute all of them ourselves.

Three outcomes decide the proposal. First, at matched total IoU, different error types should shift HOTA sub-metrics differently, with boundary leakage reducing AssA more for mask-pooling than for box pooling. Second, sensitivity curves fit on injected noise should predict the real silver-versus-gold gap within seed-level confidence. Third, profile-guided filtering of KITTI Synth should beat random filtering at equal data size. Failure of the first check falsifies the core claim. Failure of the second means the profile describes the data but does not predict tracker behavior. Failure of the third means it predicts but is not useful enough to act on. Each outcome is worth publishing as a quantitative transparency result for mined tracking data.