Many drone and field-robot trackers pair two components. A slow deep-network object detector recognizes the target reliably but has no memory from one frame to the next. A fast classical motion tracker, such as a correlation filter that slides a learned template across the image, follows pixels smoothly but has no idea what it is following. Hand-tuned rules decide which of the two to trust on each frame, for example "if the tracker's match score falls below 0.45, call the detector." We propose to treat that hand-off as a selective prediction problem: a model that may decline to answer when it is likely wrong, with a user-chosen error budget. We also propose a specific signal for deciding when to decline. That signal is the geometric disagreement between where the detector places the target and where physics-based motion propagation places it.
The bet is that this disagreement is a better warning of failure than any single model's confidence, and that the advantage is largest where it matters most: when a system calibrated on clean footage meets glare, blur, and clutter in the field. Visual hazards tend to break the two components in different ways. Blur and fast motion hurt the correlation filter's template match. Glare and unusual appearance make the detector confidently hallucinate. Two unlike sources that disagree are therefore an informative, nearly free, two-member ensemble. When the system does decline, the fallback is not a human, which is impossible at flight speeds. It is the motion model's short-horizon prediction, together with a "target uncertain" flag that the downstream controller can act on.
Core Claim
Frame-level selection scores built from one model's confidence degrade under laboratory-to-field shift. Selective Classification Under Distribution Shifts shows that softmax confidence is fragile under covariate shift because it depends on logit scale. What Does It Take to Build a Performant Selective Classifier? shows that no monotone recalibration, such as temperature scaling, can fix the resulting misordering of good and bad predictions. The same paper identifies diversity, as in ensembles, as one of the few ways to reduce that ranking error.
We claim that in hybrid tracking, the classical tracker supplies that diversity at almost no compute cost. Concretely, we define a selection score on each frame $t$ as
$$s_t = h\big(m^{\text{det}}_t,\ r^{\text{trk}}_t,\ a_t\big), \qquad a_t = \mathrm{IoU}\big(b^{\text{det}}_t,\ b^{\text{mot}}_t\big).$$
The four inputs are defined as follows:
- $m^{\text{det}}_t$ is the detector's logit margin, the scale-invariant confidence from the distribution-shift paper above.
- $r^{\text{trk}}_t$ is the correlation filter's peak-to-sidelobe response.
- $b^{\text{det}}_t$ is the detector's box.
- $b^{\text{mot}}_t$ is the box obtained by propagating the last trusted box with camera-motion-compensated optical flow and a constant-velocity filter, as in Prediction-Tracking-Segmentation.
The function $h$ is a small learned combiner. The system outputs the fused box when $s_t$ clears a threshold $\tau$. Otherwise it outputs the motion prediction flagged as uncertain, and it freezes any online model updates.
The threshold is set with the Selection with Guaranteed Risk procedure of Selective Classification for Deep Neural Networks. That procedure uses a binomial tail bound on held-out data to guarantee that the error rate among accepted frames stays below a budget $r^*$ with probability at least $1-\delta$. Here a frame counts as an error when the output box has IoU below 0.5 with ground truth.
Frames within a video are strongly correlated, so the independence assumption behind that bound fails. We therefore calibrate on blocks, one pooled estimate per sequence segment, and treat the bound's validity under shift as an empirical question rather than a promise.
Where the Pieces Come From
Three prior lines of work motivate the problem and two supply the method.
The hybrid trackers establish the architecture and expose its weak point. A Hybrid Method for Tracking of Objects by UAVs pairs YOLOv3 with a correlation filter under a fixed 0.45 response threshold. Online Learning-Based Hybrid Tracking Method for Unmanned Aerial Vehicles replaces the rule with an online AdaBoost "integrator" and reports that bad samples fed back into it corrupt the target model. Detect+Track uses optical flow to stabilize detections of deforming fish. None of the three states or checks an error guarantee, and none uses cross-source agreement as a confidence signal. That is the contrast with our proposal.
The selective prediction work supplies the method. It comprises the guaranteed-risk calibration and margin scores above, plus Calibrated Selective Classification, which trains a separate selector under simulated shifts. That work is static classification with deferral to humans. We move it to online box prediction, where abstaining triggers a mechanical fallback.
WildDash constrains the evaluation. Its collapse from 64 to 16.5 mIoU under real hazards, and the WildDash authors' warning that synthetic corruptions miss real optics, mean we cannot rely on synthetic corruption alone. We therefore split real benchmarks by hazard type.
Feature fusion enters in a narrow, testable way. The fusion question is how to combine the heterogeneous reliability cues in $h$. It is not about merging deep feature maps. Cross-modulated Attention Transformer for RGBT Tracking shows that letting one source's correlation map modulate another's helps when one source is degraded. Its authors note this fails when both degrade. That failure is the case where disagreement should drop, and our selector should decline.
We therefore compare three versions of $h$: concatenation of the three cues into a logistic model, a multiplicative gating form where agreement $a_t$ modulates the detector margin, and a two-layer network. We commit to the simplest one that performs within noise of the best.
Research Program
We will build the hybrid pipeline twice. The first build pairs YOLOv3-Tiny with a kernelized correlation filter, matching the Saribas system. The second pairs YOLOv4 with MedianFlow, matching the Son system. Both builds share the same motion-propagation module and selector. Everything runs on a Jetson-class board so that frame rate is measured under the same budget the original authors faced.
The selector adds three scalars and a tiny model. Its cost should be negligible next to the detector, and we will report it.
A second, smaller thread tests the contamination fix. Where the pipeline performs online updates, as with the AdaBoost integrator or the correlation filter's template refresh, updates are allowed only on accepted frames. We then measure whether the unrecoverable failures that Online Learning-Based Hybrid Tracking Method for Unmanned Aerial Vehicles describes become rarer.
Baselines
The naive reference is the Saribas-style hybrid with its published fixed threshold of 0.45, with no selector and no abstention. We run it at full coverage and also read it as a selective system by sweeping the threshold. Its reported result is 0.561 Success and 0.773 Precision on its custom UAV dataset (A Hybrid Method for Tracking of Objects by UAVs). On UAV123, the file reports only that it won 9 of 12 attributes. We found no reported per-attribute UAV123 success value on Lacuna, so we will reproduce the system ourselves.
The strongest existing hybrid is the online AdaBoost integrator of Online Learning-Based Hybrid Tracking Method for Unmanned Aerial Vehicles. It is the closest competitor because it already learns the arbitration instead of hard-coding it. Its reported results on its custom outdoor dataset are 0.747 Success, 0.836 Precision, and 21.90 FPS, against 0.618 for YOLOv4 alone. We run it with the same detector weights, the same calibration sequences, and an equal hyperparameter search budget, on the same hardware.
The strongest existing selective method is the single-model margin score $m^{\text{det}}_t$ with guaranteed-risk calibration. This is exactly what Selective Classification Under Distribution Shifts recommends. We found no reported tracking result for it on Lacuna. It isolates whether the tracker-derived terms add anything.
The key ablation replaces the classical tracker with a second deep tracker of similar cost, Real-Time MDNet, which reports 65.0% AUC on OTB2015. Agreement is then computed between two deep models. If heterogeneous disagreement is what matters, this homogeneous pair should rank errors worse under hazards even though it is individually stronger.
We count the proposal as beating a baseline under three conditions, all on hazard-held-out test sequences. First, lower area under the risk–coverage curve, averaged over seeds and outside the sequence-bootstrap interval. Second, a lower rate of violating the $r^*$ budget. Third, no loss of full-coverage Success and no more than a 10% frame-rate cost.
Why not just…?
Why not just tune the hand-off threshold on validation data? This is the most serious threat. Guaranteed-risk calibration is, mechanically, principled threshold tuning. If the old correlation-response score, tuned on clean sequences, keeps its error budget on hazard sequences, our proposal adds little. The deciding experiment is the attribute-held-out split. We calibrate on UAV123 sequences without illumination-variation, fast-motion, or low-resolution tags and test on sequences with them. We then compare budget-violation rates. The Calibration Trap result predicts that retuning a single monotone score cannot fix misordering, but that is a prediction to check, not a settled fact for trackers.
Why not just ensemble several detectors for disagreement? The Calibration Trap paper endorses deep ensembles as a way to reduce ranking error, so they may well rank errors as well as our signal does. Our argument is cost, plus the claim that similar networks share failure modes under the same hazard. We will run a three-detector ensemble with a matched calibration protocol and report risk–coverage against frame rate. If the ensemble wins on risk at an affordable frame rate, we will say so.
Why not just run a lightweight detector every frame with a margin score? This removes the tracker entirely. It loses temporal memory, as in YOLOv4's 0.618 against the hybrid's 0.747 in Online Learning-Based Hybrid Tracking Method for Unmanned Aerial Vehicles. It also has nothing to fall back on when it declines. It appears in our study as the single-model selective baseline, compared on both selective risk and continuity, meaning frames with no usable output.
Why not just train the selector on synthetic corruptions, as in Calibrated Selective Classification? We will include this as a variant of $h$. WildDash and the robustness literature it draws on suggest that corruptions of the AugMix kind miss real optics. Comparing the variant on real hazard splits against a selector trained only on clean data answers the question directly.
Proposed Experiments
The main benchmark is UAV123 with the attribute-held-out protocol above. The custom UAV datasets are used where available for comparison with published numbers.
To test a second domain with different hazards, we use RGBT234 and VTUAV RGB streams, with sequences split by low illumination and thermal crossover. CAFormer's 66.4% and 76.2% Success give the ceiling for a heavy fused model on these sets. We also test on the Picasso triggerfish videos, where water ripples and body deformation stress the motion component. We will report whether the method's advantage holds or reverses there, which can happen if motion propagation itself becomes unreliable.
Metrics are per-frame selective risk at fixed coverage, area under the risk–coverage curve, and the fraction of test blocks whose realized risk exceeds $r^*$ at $\delta = 0.05$. We also report full-coverage Success and Precision, the count of unrecoverable losses (tracks never re-acquired), and Jetson frame rate.
The decisive check is simple. On hazard-held-out sequences, the disagreement-based score must lower both area under the risk–coverage curve and budget violations relative to the tuned-threshold and margin-only baselines, and the homogeneous-pair ablation must do worse than the heterogeneous one. If tuned thresholds match it within the bootstrap interval, the central claim is false, and the contribution shrinks to a protocol for measuring guarantee violations in trackers.
References
- Selective Classification Under Distribution Shifts
- What Does It Take to Build a Performant Selective Classifier?
- Prediction-Tracking-Segmentation
- Selective Classification for Deep Neural Networks
- A Hybrid Method for Tracking of Objects by UAVs
- Online Learning-Based Hybrid Tracking Method for Unmanned Aerial Vehicles
- Detect+Track: robust and flexible software tools for improved tracking and behavioural analysis of fish
- Calibrated Selective Classification
- WildDash - Creating Hazard-Aware Benchmarks
- Cross-modulated Attention Transformer for RGBT Tracking
- Real-Time MDNet