Stop Denoising Where the Label Has Settled: Per-Cell Halting for Diffusion-Based Sensor Fusion

Self-driving cars fuse camera and LiDAR into a single top-down map of the scene, called a bird's-eye view (BEV) grid. The most robust recent fusion method builds this map with a diffusion model, a generator that starts from noise and cleans it up over many steps. It survives sensor failures that break single-pass methods, but it runs at about 8 frames per second, too slow for driving. One reason is that it spends the same number of cleanup steps on every cell of the map: empty road, a blinded camera cone, and a half-occluded cyclist all cost the same. This note proposes letting each cell stop being refined once the model's predicted label for that cell stops changing. Each cell carries a probability over classes such as drivable surface, crosswalk, or vehicle.

The obstacle, and the reason this needs probabilistic labels, is that "the prediction stopped changing" only means "the cell is resolved" if the model can say settled but genuinely uncertain. A model trained on hard one-hot labels cannot. On truly ambiguous cells, such as cells cut by a lane boundary, far-field sparse returns, or sensor-dead regions, it either oscillates and burns compute forever or snaps to an overconfident answer and exits too early. If the fusion model is instead trained to match soft label distributions, ambiguous cells can converge to a stable high-entropy prediction. A simple stability test then becomes a usable halting rule. The claim is that soft-label supervision is what makes label-stability halting work in diffusion fusion, and that the combination gives a better accuracy-per-compute trade-off than uniform step reduction or variance-based spatial schedules.

Core Claim

Take a diffusion fusion backbone in the style of DifFUSER and train its BEV heads with KL divergence against per-cell soft label distributions rather than hard masks. Let $\hat{\mathbf{p}}_i^{(t)}$ be the class distribution predicted for BEV cell $i$ after denoising step $t$. Cell $i$ is frozen, meaning its features are cached and it is excluded from further computation, at the first step where

$$ \mathrm{JS}\big(\hat{\mathbf{p}}_i^{(t)} \,\|\, \hat{\mathbf{p}}_i^{(t-1)}\big) < \tau , $$

where JS is the Jensen–Shannon divergence between consecutive predictions and $\tau$ is a single threshold chosen on validation data. We predict three things. First, under soft-label training this rule freezes most cells within one or two steps while keeping full-step accuracy. Second, under hard-label training the same rule either loses accuracy at matched compute or freezes far fewer cells. Third, the cells that stay active line up with cross-modal disagreement, such as camera-only regions at night or LiDAR-sparse far field, without any hand-built sensor-reliability map.

Relation to Prior Work

Two papers supply the mechanism. DifFUSER treats BEV fusion as conditional latent denoising, with gated conditioning on each modality and progressive sensor-dropout training. It reports 70.04% mIoU on nuScenes map segmentation and 73.8 NDS at about 8 FPS. It denoises the whole grid for a fixed number of steps and trains deterministic heads on hard targets, and both of those choices are what we change. Dynamic Time-Spatial Sampling shows in image super-resolution that smooth regions stabilize earlier than textured ones. It routes fewer steps to low-variance pixels, matching 15-step quality in 7 steps. Its spatial signal is local pixel variance, a property of the input. Ours is the stability of the output label distribution, a property of the task. The difference matters in BEV grids: a perfectly uniform blinded region has low variance yet is exactly where cross-modal ambiguity lives.

Two papers motivate the soft-label ingredient. Learning an Uncertainty-Aware Object Detector for Autonomous Driving replaces hard regression targets with KL divergence against label distributions, scaled by agreement between the annotated box and a multi-sweep LiDAR hull. It reports gains on ATG4D, for example bikes improving from 61.93% to 64.82% AP. We reuse its box-uncertainty recipe for the detection head. TrainRef and Moment Distributionally Robust Probabilistic Supervised Learning argue that hard targets and fixed softmax links hurt calibration under ambiguity. They are evidence that soft supervision changes what the network represents about uncertainty, not just its accuracy. Neither touches iterative inference. The early-exit literature, such as Early Exit Ensembles for Uncertainty Quantification and Hold That Exit, exits across layers per input using confidence. We exit across denoising steps per spatial cell using stability, and we argue confidence is the wrong signal for irreducibly ambiguous cells.

Research Program

The first component is cheap soft BEV labels, since repeated human annotation is unaffordable. For map segmentation, each cell's target is the fraction of the cell's area covered by each rasterized map polygon. Boundary cells then become genuinely mixed distributions rather than arbitrarily assigned ones. On top of this we add a small spatial smoothing whose width is tied to localization error, ablated separately. For detection, box targets follow the Laplace-scale recipe from the uncertainty-aware detector. All targets are derived from existing nuScenes annotations at no extra labeling cost.

The second component is training. We retrain the DifFUSER-style backbone single-task on map segmentation first, which avoids the negative-transfer confound DifFUSER itself reports for joint training. We train two otherwise identical models, one with hard-mask cross-entropy and one with per-cell KL against the soft targets. The sensor-dropout schedule is kept identical in both. A light auxiliary loss applies the segmentation head at intermediate denoising steps, so that $\hat{\mathbf{p}}_i^{(t)}$ is meaningful before the final step. This auxiliary loss is given to both models.

The third component is per-cell halting with real sparsity. Frozen cells keep cached features that still serve as context for their neighbours. Active cells are processed with sparse convolution over the active set, the same machinery LiDAR backbones already use. Because FLOP savings and wall-clock savings can diverge on GPUs, we report both, and treat wall-clock as the decisive measure.

The fourth component is analysis. We record, per cell, the step at which it froze, and relate it to LiDAR point density, camera visibility, distance, and sensor-dropout masks. We also measure per-cell calibration (expected calibration error) separately for frozen-early, frozen-late, and never-frozen cells. This tests the safety concern that a model trained to fill in missing sensors invents confident structure: settled-uncertain cells in sensor-dead regions should be well calibrated, not confidently wrong.

Baselines

The naive reference point is full-step DifFUSER-style inference trained with hard labels, using the published configuration. The reported numbers are 70.04% mIoU on nuScenes BEV map segmentation, 73.8 NDS, and about 8 FPS (DifFUSER). We retrain it ourselves on identical data and hardware so that its speed and accuracy are measured, not quoted. Our method beats it if it stays within 0.5 mIoU while using at most 40% of its per-frame denoising FLOPs and running measurably faster in wall-clock.

The strongest efficiency competitor is the same model with uniformly fewer DDIM steps, run at every step count from one to full. This is the obvious alternative and the one most likely to kill the idea. It gets the same training and tuning budget. Beating it means our accuracy-versus-compute curve lies above the uniform-step curve at matched FLOPs and matched wall-clock, not only at one cherry-picked point.

The strongest prior spatial method is the variance-based spatial schedule from Dynamic Time-Spatial Sampling, ported to BEV features with local feature variance as the routing map. We tune its thresholds on the same validation split with the same number of trials as ours. It is training-free and so cheaper to adopt, and our method must beat it at matched compute to justify retraining.

The strongest deterministic competitor is BEVFusion from Liu et al., which reports 62.7% mIoU for map segmentation and 70.2% mAP for detection in multi-task mode on nuScenes (BEVFusion: Multi-Task Multi-Sensor Fusion). It is far faster thanks to its 2 ms BEV pooling. We retrain it single-task with the same sensor-dropout augmentation and the same soft-label loss, so that the comparison isolates the value of iterative denoising.

Two ablations isolate the key ingredient. Hard-label training with the same JS halting rule tests whether soft labels are what make halting work. Soft-label training with entropy-based halting tests whether stability rather than confidence is the right signal.

Why not just…?

Why not just use a fast deterministic fuser trained with sensor dropout? BEVFusion is dramatically cheaper. Liang et al.'s BEVFusion already survives LiDAR malfunctions by keeping camera BEV independent. DifFUSER's 7.3% relative mIoU gain over BEVFusion is the only evidence in favor of diffusion here, and that comparison did not give BEVFusion soft labels or matched dropout training. This is a real threat. The retrained BEVFusion baseline settles it. If it matches our halted model under camera-drop, LiDAR-drop, and night subsets, the honest conclusion is that diffusion fusion is not worth its cost, and we will report that.

Why not just use fewer steps everywhere, or distill to one or two steps? Uniform step reduction needs no new machinery. Consistency-style methods (Jiahui Zhang et al.) and flow matching (Pan et al.) target one to four steps globally. If DifFUSER's accuracy is already flat at two uniform steps, per-cell halting has nothing to save. The first experiment therefore measures the uniform step-accuracy curve before anything else is built. Distillation is complementary rather than exclusive: halting can be applied on top of a few-step student. We test that combination if the uniform curve shows the step budget is already small.

Why not just port the variance-based spatial schedule unchanged? It is training-free and has worked for super-resolution. Our argument against it is specific: in BEV fusion, low feature variance often marks sensor-dead or empty regions, which conflate "nothing there" with "cannot see." The decisive check is to stratify both methods' errors by sensor-dropout region and night frames. If variance routing matches ours there, the label-stability signal adds nothing.

Why not just halt on confidence with a temperature-calibrated model? It is simpler. However, a confidence threshold never fires on cells whose correct answer is a mixed distribution. Temperature scaling learns one scalar on in-distribution data, while SHIFT shows segmentation mIoU falling from 83.6% to 42.8% from day to night. The entropy-halting ablation measures this directly.

Proposed Experiments

The primary benchmark is nuScenes BEV map segmentation (nuScenes), with mIoU, per-cell expected calibration error, denoising FLOPs per frame, and wall-clock FPS on one GPU. Robustness splits are camera-dropped, LiDAR-dropped, and night frames; nuScenes reports vision mAP falling 55–58% at night. Detection with soft box targets follows as a secondary task, reported in NDS, after the single-task result is in. SHIFT provides continuous fog and day-to-night transitions with RGB and LiDAR. On SHIFT we track how the fraction of active cells varies across a transition. Compute should rise smoothly as visibility degrades and fall when it recovers. No reported result was found on Lacuna for diffusion fusion on SHIFT, so all SHIFT numbers will be our own.

Three checks decide the project. First, the soft-label halted model must dominate the uniform-step curve at matched wall-clock on nuScenes. Second, the hard-label halting ablation must be clearly worse at matched compute; if it is not, soft labels are not the reason halting works and the core claim fails. Third, frozen-step maps must correlate with cross-modal disagreement more strongly than with feature variance. If all three hold, we will have shown that probabilistic supervision turns a diffusion fuser's per-cell computation into a readable measure of where the sensors disagree, while making it cheaper.