Sensor-Reliability-Aware Tracking: Spending Multi-Hypothesis Compute Where Fusion Is Failing

Self-driving perception stacks combine cameras and LiDAR (a laser sensor that measures 3D geometry) to detect objects. They then hand those detections to a tracker that links objects across time, and the tracker hands tracks to a model that forecasts where each object will go. Robust fusion research has made the detector much better at surviving a failed sensor. Its outputs to the tracker, however, are still plain boxes and confidence scores. The tracker never learns that a given pedestrian was seen only by a degraded camera because the LiDAR had dropped out. That missing information matters. When tracking errors such as identity swaps are passed to a forecasting model, forecast error is 28 times higher than when the forecaster sees ground-truth histories (MTP).

We propose to export one signal from the fusion detector and use it downstream: a per-object record of which sensor the estimate came from and how good that sensor's input looked before decoding. The tracker uses this record in two ways. It widens its matching uncertainty for objects with unreliable sensor support, and it spends an expensive "keep several association hypotheses alive" procedure only on those objects instead of on every object. The bet is that this record predicts downstream identity errors much better than detection confidence does. If it does, the stack can approach the forecasting accuracy of full multi-hypothesis tracking at a fraction of its memory and latency, which is what makes the approach usable in real time.

Core Claim

The fusion literature has found that output confidence is a poor guide when a sensor fails. MoME (Resilient Sensor Fusion Under Adverse Sensor Failures via Multi-Modal Expert Fusion) documents a "calibration trap": decoders fed by a malfunctioning sensor can produce high-confidence wrong boxes. MoME therefore routes each object query to a LiDAR, camera, or fused expert using a measure of local feature quality computed before the box is decoded.

Our claim is that this pre-decoding signal, which we call the reliability record, is also the right input for allocating effort downstream. The record contains the chosen expert, the quality score of each sensor's local features, and how those values have changed over recent frames. Concretely, we hypothesize three things. First, under sensor corruption, the record predicts which tracks will suffer an identity switch or fragmentation within the next $k$ frames substantially better than detection score does. Second, giving extra association hypotheses only to the tracks the record flags recovers most of the forecasting gain of uniform multi-hypothesis tracking. Third, the result holds at a fixed, small compute budget.

Relation to Prior Work

Prior work plays three roles here: some papers motivate the proposal, some supply methods we reuse, and some constrain the design.

Motivating work. MTP shows that propagating hard tracking decisions is the dominant source of forecasting error. It fixes this by keeping the $H$ best association hypotheses (Murty's algorithm) and running the forecaster on all of them, improving prediction by 34.2%. The cost is about 1.8GB of GPU memory per hypothesis, and MTP treats detections as fixed inputs from an unspecified detector. We keep MTP's mechanism but make $H$ a per-track quantity driven by upstream sensor state, which MTP has no access to. CoPEM (Cooperative Perception Error Models) motivates the compute constraint: in its simulated stack, pipeline delay beyond 1.0s drops safety success from 99.8% to 64.6%, below onboard-only sensing. Its experiments use synthetic object lists, not raw sensors.

Methods we reuse. MoME supplies the reliability record. Its router adds about 4ms of latency. Cocoon (Robust Multi-Modal Perception with Uncertainty-Aware Sensor Fusion) computes richer conformal reliability weights but adds about 0.5s. We therefore treat Cocoon as an alternative signal source rather than the default. Probabilistic 3D Multi-Modal Multi-Object Tracking (paper) learns a matching distance that mixes geometric uncertainty with fused appearance features. It reports 68.7 AMOTA (average multi-object tracking accuracy) on nuScenes. It uses fused features to decide what to match, but never asks whether a sensor behind those features is failing. We add exactly that input.

Constraining work. UniAD (Planning-oriented Autonomous Driving) removes hard interfaces by passing continuous queries end to end. That design could carry corrupted sensor information silently into the planner. It is also camera-only and runs at about 1.8 frames per second. We deliberately keep a modular stack with an explicit, inspectable reliability channel. The same paper notes negative transfer between perception and planning heads, which is one reason we do not retrain the detector jointly with the tracker. The same paper also reports that fused appearance features did not help pedestrian tracking, so we report pedestrians separately and expect them to be the hardest case.

Research Program

Detector and record. We train a MoME-style detector on nuScenes (nuScenes). For each detected object at frame $t$, we export the expert index, the per-sensor feature-quality scores from MoME's local attention mask, and the detection box and score. The tracker accumulates these over the object's history into a short reliability trace.

Calibration step. On a held-out split with sequence-level corruptions, we fit a small model that maps the reliability trace to $p_i$. Here $p_i$ is the probability that the greedy best association for track $i$ is wrong within the next $k$ frames. It is supervised by identity switches and fragments measured against ground truth.

Two uses of $p_i$. First, $p_i$ inflates the state covariance used in association, so that weakly supported tracks coast longer instead of being terminated and re-spawned. Second, it allocates hypotheses under a total budget $B$:

$$\max_{\{H_i\}} \sum_i p_i \, g(H_i) \quad \text{s.t.} \quad \sum_i H_i \le B,\; H_i \ge 1,$$

Here $H_i$ is the number of hypotheses kept for track $i$, and $g(H)$ is the empirically measured fraction of association errors recovered with $H$ hypotheses, estimated once from uniform-$H$ MTP runs. Because $g$ is concave in practice, a greedy allocation that gives the next hypothesis to the track with the largest $p_i[g(H_i+1)-g(H_i)]$ is sufficient. MTP's forecaster and clustering then run unchanged on the allocated hypotheses.

Corruption protocol. The nuScenes-R and nuScenes-C corruptions are defined per frame. We apply them as temporally coherent episodes, such as a LiDAR drop lasting several seconds, intermittent camera occlusion ("flicker"), or night and fog segments. This matters because propagation of errors across frames only shows up in sequences. We also run the clean setting to confirm that the method costs nothing when sensors are healthy.

Metrics. Tracking is scored with AMOTA, identity switches (IDS), and fragments (FRAG). Forecasting is scored with minADE and minFDE (average and final displacement error of the best of several predicted trajectories), together with their ratio to ground-truth-history forecasting, which mirrors MTP's 28x measurement. We also report an error propagation rate in the sense of ReCAPA (paper): the probability that a corrupted frame at time $T$ yields a forecasting failure at $T+k$. Cost is reported as GPU memory and end-to-end latency per frame. All metrics are broken down by class, with pedestrians separated.

Baselines

Naive reference point. The MoME-style detector feeds a standard greedy tracker using the learned association of Probabilistic 3DMOT, followed by MTP's forecaster with $H=1$. This is the "do nothing" stack and defines the propagation ratio we aim to shrink. On clean nuScenes the tracker's reported figure is 68.7 AMOTA (Probabilistic 3DMOT). We found no reported result on Lacuna for tracking or forecasting under nuScenes-C or nuScenes-R corruption, so this baseline must be measured by us.

Strongest existing method. Uniform MTP with a fixed $H$ for every track, run at $H=10$, the setting MTP reports at about 24 FPS, and at our budget-matched average $H$. This is the strongest competitor because it already attacks the same failure: hard association errors propagating into forecasts. MTP reports a 34.2% overall prediction improvement and up to about 70% in identity-switch scenarios, on its own inputs. We run it on our detector outputs with identical corruption episodes, the same forecaster weights, and the same tuning budget for any thresholds. Beating it means one of two things. Either we match its minADE within noise at no more than one third of its average hypothesis count and memory, or we exceed its minADE at equal budget under corruption.

Confidence-budgeted MTP. This uses the identical allocation rule, but $p_i$ is fit from detection score and box-variance history alone. It isolates the key ingredient, the pre-decoding reliability record. If it matches our method, the reliability record adds nothing.

Oracle. This uses the true corruption mask per object, meaning which sensor was actually corrupted, in place of the record. It bounds what any reliability signal could achieve.

Ablations. We separate covariance inflation from hypothesis budgeting. We also swap the upstream detector: BEVFusion (paper), whose fusion attention weights would serve as a weaker record, and Cocoon conformal scores used as the record despite their latency.

Why not just…?

Why not just use a more robust detector and stop there? MoME already raises mAP by 4.2 points under LiDAR drop. A single-decoder baseline in the same paper falls from 72.9 to 44.7 NDS (nuScenes detection score). Better detection could plausibly remove most downstream failures, and this is a real threat. The naive baseline with the MoME detector measures it directly. If the propagation ratio under corruption is already near the clean-condition ratio, the proposal is unnecessary and we will report that. Our expectation runs the other way. Even a robust detector produces intermittent boxes during sensor flicker, and MTP shows that intermittency, not raw box accuracy, drives the forecasting blow-up.

Why not just run uniform multi-hypothesis tracking everywhere? It works, but memory and latency grow linearly with $H$ (about 1.8GB per hypothesis). CoPEM's latency cliff makes this overhead a safety issue rather than a mere cost. If a small uniform $H$, say 2 or 3, already recovers nearly all of the gain, then selective allocation is pointless. The budget-matched uniform baseline settles this question.

Why not just let detection confidence decide where to spend hypotheses? MoME's calibration trap is direct evidence that confidence stays high when a sensor is failing. That is exactly the case in which the tracker most needs help. The confidence-budgeted baseline tests whether the trap also holds at the tracking level.

Why not go fully end-to-end like UniAD? UniAD reports large planning gains, but it is camera-only and runs at about 1.8 FPS. Its soft queries also give no handle for checking whether corrupted sensor input is flowing downstream. Adapting it to LiDAR fusion under corruption is a separate project. We do not claim our approach dominates it. We claim a cheaper, inspectable alternative for modular stacks.

Decisive Checks

The first check gates the project. On corrupted validation sequences, the reliability record must beat detection score at predicting identity switches and fragments in the next $k$ frames, measured by area under the ROC curve. A meaningful gap is required. If there is none, the allocation idea has no foundation and we stop. The second check is the budget curve: minADE against average hypotheses per track for uniform MTP, confidence-budgeted MTP, our method, and the oracle. Success means our curve sits close to the oracle and clearly above the confidence-budgeted curve at low budgets. The third check is the clean-sensor control: the method must not degrade AMOTA or minADE when no corruption is present. The fourth is the pedestrian breakdown. We report pedestrians honestly, since prior multi-modal tracking found no pedestrian gains, and that result will tell us whether the reliability record helps the objects that most need it.