A self-driving car's 3D detector usually combines a laser scanner (LiDAR) with cameras. It fails badly in a specific way: when the LiDAR is blocked or returns nothing from dark or rain-soaked objects, accuracy collapses toward zero instead of falling back to what the cameras alone could do. Methods that estimate how much to trust each sensor, by reading signal-to-noise, entropy or illumination off the input, have trouble with structured damage such as mud on a lens, because mud looks like valid image content. We propose estimating trust from a different signal. Diffusion-based detectors start from random boxes and refine them over several steps. If each sensor proposes its own correction at every step, a sensor that is corrupted, or too sparse on a distant object, will propose corrections that wander from step to step and disagree with the other sensor. The trajectory itself then shows which sensor is helping, box by box. The same signal also shows when a box has stopped moving, so refinement can stop early.
Put plainly, we turn the main weakness of generative detectors, their many sequential refinement steps, into the source of a per-object, per-step reliability estimate. That estimate does two jobs: it re-weights LiDAR against camera, and it decides when each box can stop refining. If it works, the detector degrades gracefully under sensor failures it was never trained on, and it spends its slow refinement steps only on the hard, sparse, occluded objects that need them.
Core Claim
Generative object localization treats detection as denoising. A diffusion model, which learns to turn random noise into samples step by step, is applied to box coordinates. The model starts from Gaussian boxes and moves them toward objects, conditioned on scene features. This was introduced in 2D by DiffusionDet and carried into LiDAR by Diff3Det. The only LiDAR-camera generative detector in our file, 3DifFusionDet, fuses the two sensors with a fixed cross-attention module that is identical at every step and for every box. It reaches 91.3% Easy and 80.4% Hard mAP on KITTI at 8 steps, but runs at 0.6 FPS. It has no mechanism for noticing a degraded sensor, and its authors explicitly call for self-adaptive step counts.
Our claim is that the update a sensor proposes over refinement steps measures that sensor's reliability for that object. We expect this measure to stay informative under corruptions that input-level reliability estimators cannot recognize. Concretely, the decoder keeps two cross-attention streams, one reading LiDar bird's-eye-view features and one reading camera features. Each stream predicts its own correction to the current box, written $\Delta_L$ and $\Delta_C$. The box at step $t$, written $x_t$, is updated as
$$x_{t-1} = x_t + w_L\,\Delta_L + w_C\,\Delta_C, \qquad w_m \propto \frac{1}{\sigma_m^2 + \lambda\,\|\hat{x}_0^{L} - \hat{x}_0^{C}\|^2}.$$
Here $\hat{x}_0^{m}$ is stream $m$'s current guess of the final clean box. The term $\sigma_m^2$ is the running variance of that guess across the steps taken so far, which measures self-inconsistency. The second term in the denominator measures cross-sensor disagreement, scaled by a learned constant $\lambda$. The inverse-variance form comes from the standard rule for combining two noisy estimates. A small learned gate then corrects the weights using point count inside the box, distance and step index. A box stops refining once both streams' guesses have moved less than a threshold $\epsilon$ for $k$ consecutive steps.
Geometric incompleteness sets the regime where we expect this to matter. Line-of-sight LiDAR returns only the near shell of an object and thins out with distance (Deep Learning for LiDAR Point Clouds in Autonomous Driving: A Review). We therefore expect the LiDAR stream's guesses for far or occluded objects to wander, which should shift weight to the camera exactly where LiDAR is weakest.
Research Program
The model is a DiffusionDet-style decoder on a standard LiDAR voxel backbone and a camera backbone, as in 3DifFusionDet. We reuse Diff3Det's practice of resampling initial boxes that contain too few points, so the decoder does not waste capacity on empty space. In our version, a box is resampled only if it is empty in both LiDAR and camera frustum features. Otherwise camera-only objects would be thrown away when LiDAR fails.
Training uses sensor dropout (Liu et al.) applied per stream, so that each stream is a competent denoiser on its own. Without this, $\Delta_C$ would be meaningless whenever LiDAR is absent, and the LiDAR dependency trap would simply reappear inside our decoder. The two streams share one decoder pass and differ only in their cross-attention, so the extra cost per step is one attention block. We will measure that cost rather than assume it is small.
The analysis has three parts. First, we test whether the reliability weights track ground-truth corruption on nuScenes-R. Second, we check whether exit step correlates with distance and with point count inside the box. Third, we look for where the method breaks. The obvious breaking point is the stuck-frame corruption. A frozen camera frame produces boxes that are stable but wrong, so self-consistency alone cannot flag it, and only cross-sensor disagreement can. We will report this case separately.
Prior work plays different roles here. Benchmarking the Robustness of LiDAR-Camera Fusion motivates the work: it diagnoses the dependency trap and shows that mud is worse than a missing image. TransFusion is the strongest discriminative fusion detector and does soft attention in a single pass. Input-statistics gates such as EGM-Det (entropy), SNR maps (Kanghao Chen) and illumination gating supply the competing hypothesis. That hypothesis says reliability can be read off the input itself, and we test it directly. HCMS and the gate in 3D MeanFlow's PointPlug show that selective computation can save compute, by 44% in HCMS and by completing only the top-$K$ sparse proposals in PointPlug. Both decide once, before processing. Our exit is decided during refinement, from how the box is actually moving.
The Diff3Det result constrains the latency claim. On KITTI Hard, going from 1 to 4 steps raised AP only from 76.07% to 76.44%. Extra steps may therefore help little on average. Our bet is that their value is concentrated in corrupted and far-range cases, and the experiments test this directly.
Baselines
The naive reference is a static-fusion generative detector. This is our own decoder with one fused cross-attention stream, a fixed step count and no reliability weights, essentially a reimplementation of 3DifFusionDet on our backbones. It shows what the generative formulation gives before any gating. We will run it at $D=1$, $D=4$ and $D=8$ to trace its speed-accuracy curve. For reference, the reported KITTI figures are 91.3% Easy and 80.4% Hard mAP at $D=8$ and 0.6 FPS (3DifFusionDet).
The strongest existing method is TransFusion, a discriminative transformer detector with spatially modulated soft cross-attention. It was the most robust model on nuScenes-R and reports 68.9% mAP and 71.7% NDS on the nuScenes test leaderboard (TransFusion). We found no reported per-corruption scalar result on Lacuna for TransFusion on nuScenes-R; that paper reports only qualitative findings. We will therefore rerun TransFusion ourselves on nuScenes-R with the released corruption toolkit. TransFusion will also be given the same robust fine-tuning on toolkit corruptions and the same sensor-dropout schedule as our model, with an equal number of training epochs and the same validation-based tuning budget. Without this, any robustness gap could come from augmentation rather than from architecture.
We will also report single-sensor references: CenterPoint for LiDAR and DETR3D for camera, both as evaluated in the benchmark paper, plus our own model with one stream disabled. The camera-only number sets the "graceful fall-back" target.
Two ablations isolate the key ingredient. The first replaces trajectory-derived weights with an input-statistics gate, using image entropy and point density per box, inside the identical decoder. The second keeps dual streams and sensor dropout but fixes the weights at 0.5. We count the method as beating its competitors only under three conditions. It must match robust-fine-tuned TransFusion on clean nuScenes, within one mAP point. It must exceed it on held-out corruption types. And it must exceed the entropy-gate ablation specifically on lens mud and LiDAR object failure.
Why not just…?
Why not just fine-tune TransFusion on corrupted data with sensor dropout? This is the most serious threat. The benchmark paper reports that robust fine-tuning improves mean robustness without hurting clean accuracy. If corruption at test time resembles corruption at training time, this may be enough. Our argument is about corruptions the model has not seen: a learned single-pass gate must recognize what corruption looks like, whereas trajectory instability needs no such recognition. We settle this with leave-one-corruption-out training. For example, both models train on camera blackout and LiDAR field-of-view loss, then are tested on mud and object failure. If fine-tuned TransFusion matches us on held-out corruptions, the architectural claim fails. We would then report the method as a latency contribution only.
Why not just gate on input quality, as entropy-guided or SNR-based fusion does? The benchmark finding that mud hurts more than total camera loss suggests input statistics do not separate structured dirt from valid texture. A muddy patch has normal entropy, but it should produce box corrections that conflict with LiDAR. The entropy-gate ablation tests this directly with the same decoder, data and budget. If it ties our method on mud, the trajectory signal adds nothing, and we will say so.
Why not just use one or two fixed steps, or distill the box decoder to a single step as Mean Flows does for generation? Mean Flows and 3D MeanFlow show that one-step generation can be nearly as good as many steps, and Diff3Det's small 1-versus-4-step gap points the same way. If fixed $D=1$ matches adaptive exit at equal wall-clock cost, early exit is unnecessary. Even then, the reliability weights could still be computed from cross-sensor disagreement at the first step. We compare adaptive exit against fixed-step baselines at matched average numbers of decoder evaluations, broken down by distance and by corruption. The per-step costs in 3DifFusionDet are large: 0.6 FPS at 8 steps against 1.9 FPS at 1 step. Whether extra steps pay off is therefore an empirical question we answer, not one we assume.
Proposed Experiments
The main testbed is nuScenes, because it has synchronized multi-camera and LiDAR data, together with nuScenes-R, which covers LiDAR field-of-view loss, object failure, camera loss, lens mud, spatial jitter and stuck frames. Waymo-R serves as a denser-LiDAR replication. KITTI connects us to the reported generative-detector numbers: 77.90% Car Moderate AP for Diff3Det, and 3DifFusionDet's figures above. All latency is measured on one GPU type with identical backbones.
We report three groups of metrics. Accuracy is measured as mAP and NDS, overall, per corruption and for objects beyond 50 m. Robustness is a fall-back ratio: corrupted-fusion mAP divided by the clean single-sensor mAP of the surviving sensor. Latency is measured as frames per second and as mean decoder evaluations per box.
Four checks decide the proposal. First, under LiDAR object failure, the fall-back ratio must approach 1, meaning the detector is at least as good as camera-only, where existing fusion models drop toward zero. Second, under lens mud, the learned camera weight must fall about as far as it does under camera blackout. Third, on held-out corruption types, we must beat robust-fine-tuned TransFusion. Fourth, at matched compute, adaptive exit must beat the best fixed step count. For the fourth check, mean exit step should rise with distance and fall with LiDAR point count inside the box.
We expect the stuck-frame corruption to be the hardest case, and we will report it as the main failure mode if cross-sensor disagreement cannot catch stable-but-stale frames. Failing the first three checks would refute the central claim. Failing only the fourth would remove the latency contribution but keep the robustness one.
References
- DiffusionDet: Diffusion Model for Object Detection
- Diffusion-Based 3D Object Detection with Random Boxes
- 3DifFusionDet: Diffusion Model for 3D Object Detection with Robust LiDAR-Camera Fusion
- Deep Learning for LiDAR Point Clouds in Autonomous Driving: A Review
- Benchmarking the Robustness of LiDAR-Camera Fusion for 3D Object Detection
- TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers
- EGM-Det: Entropy-Guided Multimodal Adaptive Fusion for UAV RGB-IR Object Detection
- All-Day Multi-Camera Multi-Target Tracking
- HCMS: Hierarchical and Conditional Modality Selection for Efficient Video Recognition.
- 3D MeanFlow: One-Step Point Cloud Completion and Generation via Average-Velocity Transport
- Mean Flows for One-step Generative Modeling