Calibrating Detector Box Uncertainty for How Trackers Actually Use It

Object detectors that output a box together with an uncertainty, meaning a predicted spread around each box edge, are usually judged one frame at a time. In driving, though, those uncertainties are consumed by a tracker or a world model that fuses many frames and predicts where objects will be next. Such a tracker typically makes two quiet assumptions. The first is that the detector's errors in consecutive frames are independent. The second is that object motion between frames is nearly deterministic, so it carries little or no process noise. Both assumptions fail in the cases that matter. A pedestrian half-hidden behind a parked van produces the same wrong edge for twenty frames in a row. Fusing those frames as if they were independent shrinks the tracker's uncertainty roughly as one over the number of frames while the actual error stays put. The result can be a detector that is perfectly calibrated per frame and a tracker that becomes more overconfident the longer it watches.

We propose to calibrate box uncertainty where it is used: on the multi-step predictive distributions of a stochastic box-state transition model. The detector gets one small addition, a per-box estimate of how strongly its current error persists from the previous frame. It is trained jointly with a learned transition model under a differentiable calibration loss applied to k-step-ahead predictions. The claim is that this fixes a failure that per-frame calibration, post-hoc scaling, and plain likelihood training each leave in place: overconfidence that grows with horizon and depends on scene content.

Core Claim

Probabilistic box regression has well-developed per-frame machinery. Bounding Box Regression With Uncertainty for Accurate Object Detection predicts a Gaussian per edge and trains it with a loss that trades error against log-variance. Gaussian YOLOv3 does the same in a one-stage detector. Generalized Focal Loss and D-FINE replace the Gaussian with a learned histogram over edge positions.

None of these methods check whether the spreads they output are calibrated. The calibration work that does check is post-hoc and single-frame. Evaluating and Calibrating Uncertainty Prediction in Regression Tasks introduces the Expected Normalized Calibration Error (ENCE), which bins predictions by predicted spread and compares each bin's actual error to its predicted spread. It then fixes miscalibration with a single learned scale on the spread. Parametric and Multivariate Uncertainty Calibration for Regression and Object Detection fits Gaussian-process recalibrators after training, including covariance across box coordinates. It also argues that downstream Kalman-style filters need parametric means and covariances rather than arbitrary histograms.

The one in-training multivariate calibration method in the file is Calibrated Multivariate Distributional Regression with Pre-Rank Regularization. It adds a kernel-smoothed calibration penalty (PCE-KDE) on scalar projections of the error vector and shows that only a projection along the predicted covariance's principal axes catches rotated correlation structure. It was validated only on tabular data. We reuse its regularizer as our method ingredient. The difference is what we apply it to: the error vector is the stacked multi-step error of a box track, and the structure to be caught is temporal correlation rather than cross-coordinate correlation.

On the dynamics side, Stochastic Latent Actor-Critic shows that replacing a deterministic transition with a learned stochastic one prevents compounding rollout error. Its state is a pixel latent rather than object boxes, and its noise is never checked for calibration. Bench2Drive shows that ignoring stochasticity is fatal in closed loop: an open-loop regressor gets a 0% success rate. That motivates the problem but tells us nothing about where the overconfidence comes from.

Our specific, falsifiable claim has two parts. First, a detector calibrated per frame and fed to a filter with independent observation noise and near-deterministic transitions will show coverage of its nominal 95% regions that falls with track length, and falls faster under occlusion. Second, joint training with a persistence head and a horizon-level calibration loss restores coverage across horizons and under held-out towns and weather, without lowering average precision.

Research Program

The detector is a Generalized Focal Loss style dense detector, whose per-edge histogram we convert to a mean and variance by moment matching. This conversion is our answer to the parametric-versus-histogram tension: the histogram stays as the training representation, and the filter receives the moments it needs. We add two outputs per box. One is a cross-edge covariance in the spirit of the multivariate work above. The other is a persistence coefficient $\rho_t \in [0,1)$, the predicted correlation between the detector's error at frame $t$ and at frame $t-1$.

The tracker is a filter over box state $b_t$, meaning edges plus their velocities. Its transition model is a small network that outputs a mean motion and a content-dependent process-noise covariance $Q_t$. The deterministic transition assumption corresponds to forcing $Q_t = 0$.

Detector errors are treated as correlated over time. The observation error $e_t$ evolves as

$$e_t = \rho_t\, e_{t-1} + \sqrt{1-\rho_t^2}\;\epsilon_t,\qquad \epsilon_t \sim \mathcal{N}(0, \Sigma_t),$$

where $\Sigma_t$ is the detector's moment-matched covariance. The standard way to handle this is to add $e_t$ to the filter state, which keeps filtering exact and differentiable.

Training minimizes per-frame negative log-likelihood plus a horizon calibration term:

$$\mathcal{L} = \sum_t \mathrm{NLL}_t + \lambda \sum_{k \in K} \mathrm{PCE\text{-}KDE}\big(\{r_{t+k|t}\}\big).$$

Here $r_{t+k|t}$ is a scalar pre-rank of the error between the ground-truth box at $t+k$ and the filter's $k$-step prediction made from frames up to $t$. We use two pre-ranks, the Mahalanobis distance and the principal-axis projection from the pre-rank paper, and $K$ is a small set of horizons. Gradients flow through the filter into the detector's covariance and persistence outputs and into the process-noise network. The calibration loss therefore teaches each part whose uncertainty explains an error.

Ground-truth tracks come from the Bench2Drive training set (2 million frames, 13,638 clips, 23 weathers, 12 towns, 3D boxes). We project those 3D boxes into the front camera to obtain 2D tracks. Towns and weathers are held out to test shift. KITTI is used only for per-frame comparison against published ENCE numbers.

The kernel-based penalty needs many residuals per batch, which the file flags as a known weakness. We pool residuals across all boxes and horizons in a batch and keep a moving-average buffer, following Unconstrained Monotonic Calibration of Predictions in Deep Ranking Systems.

Baselines

The naive reference is the detector trained with the Gaussian likelihood of He et al. and fed to a constant-velocity filter. That filter has near-zero process noise and treats the detector's errors as independent across frames. This is the deterministic-transition configuration that most pipelines effectively use.

The strongest per-frame competitor is post-hoc recalibration. We use the single-scale method of Evaluating and Calibrating Uncertainty Prediction in Regression Tasks, which reports KITTI x-position ENCE falling from 16.5% to 8.3% with scaling and to 6.0% with an interval-based method. We also use the multivariate Gaussian-process recalibration of Parametric and Multivariate Uncertainty Calibration for Regression and Object Detection; no reported result found on Lacuna for its calibration numbers on COCO or BDD100K.

The strongest competitor on our actual question is a baseline we construct, because no paper in the file reports one. It is post-hoc horizon recalibration: freeze the trained detector and filter, then fit one covariance scale per horizon, plus a scalar process-noise level, on validation tracks. No reported result found on Lacuna for horizon-level box calibration on Bench2Drive or any video benchmark.

Every method uses the same detector backbone and training data. Every method gets the same validation split for fitting or tuning, and the same hyperparameter budget, counted in tuning runs, which is spent on $\lambda$ for our method and on recalibrator settings for the others.

Three ablations isolate the ingredients. The first removes the persistence head ($\rho_t = 0$). The second keeps the calibration term but applies it only at $k=0$, which is per-frame in-training calibration and the direct transfer of the pre-rank work to boxes. The third forces deterministic transitions ($Q_t = 0$).

Our method beats the baselines if it achieves lower ENCE and PCE at horizons of five frames and beyond, conditioned on occlusion level, on held-out towns and weathers. This must hold against the horizon-recalibration baseline in particular, with COCO-style average precision within 0.5 points of the naive detector. For scale, Generalized Focal Loss reports 45.0% AP on COCO test-dev with ResNet-101.

Why not just…?

Why not just recalibrate the detector post hoc, per frame? Per-frame scaling demonstrably fixes per-frame ENCE, and we expect it to match us at $k=0$. It cannot touch temporal correlation, because a frame-level scale has no notion of how many frames the filter has fused. Our first experiment measures exactly this: coverage versus track length for a per-frame-recalibrated detector.

Why not just tune the filter's noise post hoc, per horizon? This is the real threat. If a few global scales restore coverage, our method is unnecessary. Our bet is that persistence is content-dependent: occlusion persists while blur does not. A global scale should then over-cover easy tracks and under-cover occluded ones, which will show up as good average coverage but poor ENCE when binned by predicted spread and by occlusion. The pre-rank paper's finding that average calibration hides structural misspecification supports this bet but does not prove it. If the horizon-recalibration baseline matches us on conditional ENCE under held-out conditions, we will report that and the claim fails.

Why not just train a stochastic box-state model by likelihood alone? Likelihood is a proper scoring rule, so in principle it rewards calibration, and a SLAC-style stochastic transition on boxes might suffice. The tabular evidence in Calibrated Multivariate Distributional Regression with Pre-Rank Regularization is that likelihood-trained mixtures stay measurably miscalibrated and that the penalty reduces calibration error without hurting likelihood. The $\lambda = 0$ ablation tests whether this holds for box tracks.

Why not just use ensembles or Monte Carlo dropout? Probabilistic Object Detection found Monte Carlo dropout still spatially overconfident. These methods also multiply inference cost, and they estimate per-frame model uncertainty rather than the persistence of the detector's errors over time. We include a five-member ensemble at $k=0$ only as a sanity check.

Proposed Experiments

The first and decisive check is diagnostic. On held-out Bench2Drive clips, we take the naive detector after per-frame recalibration and plot 95% coverage and ENCE against horizon and track length, split by occlusion. If coverage does not degrade with track length, the premise is wrong. The project would then reduce to per-frame in-training calibration, the $k=0$ ablation, which we would report as the main result.

The second check pits the full method against post-hoc horizon recalibration on held-out towns and weathers. We report conditional ENCE, QCE, Mahalanobis and principal-axis PCE, and negative log-likelihood. The third check reports average precision and PDQ spatial quality, to show that per-frame quality is preserved. The fourth checks KITTI per-frame ENCE, so our $k=0$ numbers sit directly beside Levi et al.'s 8.3% and 6.0%.

Only if the second check succeeds will we take one downstream step. We will feed the tracks to a simple planner in Bench2Drive closed loop and compare interactive-scenario success under calibrated versus naive tracks. This tests, rather than assumes, whether horizon miscalibration contributes to the interactive failures that benchmark exposes.