Teaching Trackers How Far the Target Might Go: Soft Containment Targets for Search-Window Sizing

A visual tracker follows one object through a video. In each new frame it crops a "search window" around the last known position and looks for the target only inside it. A small window is cheap and contains few look-alike distractors, but the target can jump outside it. A large window is safe but slow, and it dilutes the target among clutter. Adaptive trackers resize the window frame by frame. The networks that choose the size, however, learn from hard labels. Each training frame is told that exactly one window size is "correct," even though the right size depends on where the target will move next, and that is genuinely uncertain. We propose to replace those hard labels with soft ones. The supervision for window size becomes a probability distribution over how far the target could plausibly travel. The tracker then learns a calibrated estimate of "the probability the target stays inside a window of this size." At test time it picks the smallest window whose containment probability meets a user-chosen risk level.

This matters most when trackers leave the data they were trained on, for example a model trained on generic video and deployed on drones or thermal cameras. A window policy that is overconfident on familiar motion will shrink at exactly the wrong moments under new motion statistics. Work on soft-label training in classification shows that preserving uncertainty in the targets makes models less overconfident and better behaved under shift. We test whether the same holds when the "label" is a spatial search extent rather than a class.

Core Claim

Search-window size is an ordered quantity, and the correct supervision for it is a cumulative distribution: for each candidate size, the probability that the target will still be inside. We will construct these soft containment targets from data without new human annotation. A tracker-side regulator trained on them with ordinal distribution losses will give containment probabilities that stay calibrated under domain shift. Hard-label regulators, uniformly smoothed labels, and post-hoc recalibration of a hard regulator will not do as well. The practical consequence is a single knob, the acceptable escape rate $\alpha$, that trades compute for robustness and still holds on unseen domains.

The closest existing work is SRRT. Its regulator outputs a softmax over a few discrete crop scales, picks the most probable one, and is supervised with categorical cross-entropy against hard pseudo-labels from a bounding-box containment heuristic. We change three things. The target is a soft ordinal distribution over required scale, not a one-hot pick. The output is read as a containment curve and thresholded at a risk level, not taken as an argmax. And the evaluation centres on out-of-domain calibration rather than in-domain accuracy alone.

Where the Soft Labels Come From

Soft-label methods normally rely on several human annotators per item: the crowd labels of A Case for Soft Loss Functions, the seven pathologists behind agreement-aware label smoothing, and the perspectivist argument of Cabitza et al. that disagreement is signal. Tracking datasets have one box per frame. The file names this cost of collecting label distributions as an open problem. We therefore build the distribution from plausible futures rather than from plural annotators.

For a training frame with motion history $h$ (recent displacements and scale changes, normalized by target size), we retrieve the $k$ training frames with the most similar histories across the whole training corpus. We record the window scale each of them would have needed to contain the target $\Delta$ frames later. We also jitter the current box within the annotation noise typical of the dataset. Write $\hat F(s \mid h)$ for the fraction of these neighbours whose required scale is at most $s$. This empirical curve is the soft target. It is monotone in $s$ by construction, and it encodes "targets moving like this one sometimes swerve."

Neighbour counts are sparse, so we will compare raw fractions with the softmax-style normalization that Uma et al. use to keep sparse vote counts from assigning zero mass. As a fallback that needs no retrieval, we use the two-pass confidence-aware smoothing of Wei et al., which uses a first model's confidence as a proxy for difficulty.

The regulator is a small head on the tracker's own features. This avoids the decoupled external network that the file flags as a limitation of SRRT. It predicts $F_\theta(s \mid x_t)$ over $S$ ordered scales. We train it with a cumulative loss of the kind shown in Reducing Supervision Uncertainty Induces Model Miscalibration to yield models calibrated at temperature 1 on ordinal Likert data. The concrete loss is the Cramér distance,

$$\mathcal{L}(\theta) = \sum_{s=1}^{S} \big(F_\theta(s \mid x_t) - \hat F(s \mid h_t)\big)^2,$$

which penalizes mass placed at the wrong distance along the scale axis, not just the wrong bin. At inference the window is the smallest $s$ with $F_\theta(s \mid x_t) \ge 1-\alpha$. Because the targets look $\Delta$ frames ahead using matched histories, the policy can widen before an abrupt manoeuvre. This addresses the one-frame reactive lag documented for velocity-driven resizing in SAMViTrack.

Research Program

The work has three parts.

The first is target construction and analysis. We build $\hat F$ on GOT-10k and LaSOT training splits and measure how much probability mass it spreads across scales as a function of motion type. If almost all frames get near-one-hot targets, the soft labels add nothing and the hypothesis fails early.

The second is training the regulator head on top of a fixed base tracker. We use one transformer tracker for generic benchmarks and the lightweight MViTrack-style backbone of SAMViTrack for UAV benchmarks. The base tracker weights are shared across all regulator variants, so differences come only from window policy.

The third is the generalizability study, which takes its framing from the small-data and shift concerns in the file. Regulators are trained on generic ground-level video and tested on UAV (DTB70, UAV123, VisDrone2018, UAVDT) and thermal anti-drone data (AntiUAV, AntiUAV410). Thermal crossover, where the target's heat signature matches the background, is a known failure mode in the Anti-UAV benchmark. We also train regulators on 5%, 20% and 100% of training trajectories. This tests whether soft targets act as the regularizer that Singh et al. found in classification, where soft labels allowed longer training without overfitting and tracked human entropy far better. That work used frozen encoders on static classification; we test whether the effect carries to a sequential, spatial decision.

The DCOM work Enhanced Bounding Box Estimation with Distribution Calibration also borrows offline statistics from similar training clips. It calibrates target features for box estimation and leaves the search window untouched. It supplies the precedent for retrieving statistics from similar clips. We apply that idea to window extent instead.

Baselines

The naive reference is the base tracker with its fixed search factor, evaluated at two factors: its default and a larger one matched in average window area to our method at $\alpha=0.05$. This separates "smarter sizing" from "simply bigger windows."

The kinematic heuristic is the speed-driven resizing of SAMViTrack, applied at inference on the same backbone. It reports 77.4% precision on the VisDrone2018 fast-motion subset, 0.6 points over MViTrack. Matching or exceeding that subset precision while lowering escape events on abrupt-reversal frames would count as beating it.

The strongest learned competitor is an SRRT-style regulator trained with hard containment pseudo-labels and argmax selection. SRRT reports that its regulator boosts TransT by 3.1% AUC and SiamRPN++ by 4.6% AUC on LaSOT, with gains of 9.2% on Fast Motion and 7.1% on Full Occlusion. The file gives only these relative gains, not SRRT's absolute LaSOT AUC, so we re-run it ourselves with an identical head, features, scale set, training data and tuning budget. To beat it, our method must achieve equal or better LaSOT AUC in domain and a lower containment calibration error out of domain. It must also either match it on UAV and thermal AUC at lower mean window area or exceed it at equal area.

On thermal anti-drone data the reference is FocusTrack. It expands its window when a learned confidence token falls below a threshold, and it reports 67.7% AUC on AntiUAV and 62.8% on AntiUAV410 at 30G MACs. We do not expect to beat its full system, which adds a background-masking module. Instead we will swap our regulator in for its threshold-based expansion rule, keep everything else fixed, and compare.

Two ablations isolate the key ingredient. The first trains the same head with uniformly smoothed hard labels, which tests whether the structure of the soft target matters or only its softness. The second uses soft targets with plain cross-entropy instead of the cumulative loss, which tests the ordinal treatment.

Why not just…?

Why not just apply post-hoc temperature scaling or conformal calibration to a hard-label regulator? This is the most serious threat. A conformal threshold on a hard regulator's scores could also deliver a target escape rate. The miscalibration study in the file argues that global temperature scaling cannot restore per-instance uncertainty once hard labels have collapsed it. A single global temperature can fix average confidence but cannot make the model widen specifically on swerving targets. Conformal guarantees also assume calibration and test data are exchangeable, which is exactly what breaks under domain shift. We will run the hard regulator with both temperature scaling and split-conformal thresholds fitted on in-domain validation data, then measure realized escape rates on UAV and thermal data. If they hold $\alpha$ as well as our method at similar window area, the soft-label claim is falsified. We would then report the simpler recipe.

Why not just use a constant-velocity Kalman prediction with its covariance to size the window? An analytic Gaussian over future position gives a soft containment curve for free. It is unimodal and its spread grows with velocity, so it cannot represent the bimodal "continues or reverses" futures that cause escapes in drone footage. Nearest-neighbour targets can represent them. We include it as a target-construction variant: the same head and loss, trained on Gaussian-derived $\hat F$. If it matches retrieval-based targets, we adopt it as the cheaper option.

Why not just use uncertainty from the box-regression head, as UATracker does? Model Uncertainty Guides Visual Object Tracking predicts aleatoric variance of the current box and uses it to filter memory updates, reporting 70.9% AUC on OTB100. That variance describes how precisely the target is localized now, not how far it will move. A sharply localized fast drone has low variance and high escape risk. We test this directly by scaling the window with UATracker-style predicted variance on the same backbone.

Proposed Experiments

In-domain evaluation uses LaSOT, GOT-10k, NfS, OTB100 and VOT2018. Shifted-domain evaluation uses DTB70, UAV123, VisDrone2018, UAVDT, WebUAV-3M, AntiUAV and AntiUAV410. We report success AUC and precision, plus three window-specific measures. The first is containment rate: the fraction of frames where the target is inside the chosen window. The second is mean window area together with measured FPS and MACs. The third is the expected calibration error of predicted containment probability against realized containment.

The decisive check is a curve of realized escape rate against nominal $\alpha$ on shifted domains. Our claim holds if this curve stays near the diagonal for the soft-label regulator while hard, smoothed and recalibrated regulators fall below it, escaping more often than promised. The second check is the data-fraction sweep. If soft targets regularize as in classification, the gap over hard labels should widen at 5% of trajectories. The third check restricts evaluation to frames that follow abrupt direction reversals, taken from fast-motion attribute labels. On these frames, anticipatory widening should reduce escapes relative to SAMViTrack's reactive rule. If none of these gaps appears beyond seed variance across three seeds, we will conclude that the window decision does not benefit from soft supervision.