Self-driving cars combine cameras, LiDAR and radar so that when one sensor fails, the others can cover for it. The best current fusion systems route each detected object to whichever sensor stream looks healthy. They quietly assume that at most one sensor is broken at a time. Real weather breaks that assumption. Low sun can blind the camera and saturate the LiDAR at once, and rain can smear the lens while scattering laser beams. When every stream is degraded, the router still has to pick one, and nothing tells the car that there is no safe choice left. We propose a runtime monitor for this case. It runs a separate detector on each of the three sensors and tallies, over a short window of frames, how often the detectors agree or disagree about which objects exist. From those tallies alone, without ground-truth labels, it estimates each sensor's current miss rate and false-alarm rate. It then decides which subset of sensors is degraded, including the case where all of them are.
The statistical side of the problem sets the design. In any single frame only a handful of objects fall where all three sensors can see them. With so few events, a per-frame judgment is noise, much as a physics experiment that counts four events carries a 50% relative uncertainty. The monitor therefore accumulates evidence across frames, with a stated false-alarm rate and a predicted detection delay. The same counting logic separates a sensor fault from a genuine rare object such as debris or an animal. A real object makes the sensors agree more often, while a fault makes them disagree.
Core Claim
With three detectors that miss or hallucinate objects roughly independently once we know whether an object is really there, the pattern of agreement counts is enough to recover each detector's error rates. This is the classic identifiability result for latent-class models, which infer hidden labels from several noisy annotators, and it needs at least three views. Because nuScenes cars carry cameras, LiDAR and radar, we have exactly three views. Within a sliding window, let $\theta_m = (\text{miss}_m, \text{false-alarm}_m)$ be the rates of sensor $m$. The monitor fits $\theta$ by maximum likelihood to the eight-way agreement counts, which record every combination of which sensors reported a given object. It then scores each of the $2^3$ health hypotheses $H_S$, where $S$ is the set of degraded sensors, and runs a sequential likelihood-ratio test against the healthy reference. Wald's analysis of such tests gives the expected number of frames to a decision:
$$\mathbb{E}[T] \approx \frac{\log(1/\alpha)}{D(H_S \,\|\, H_\emptyset)},$$
where $\alpha$ is the tolerated false-alarm probability and $D$ is the per-frame Kullback–Leibler divergence between the agreement-count distributions under fault set $S$ and under the healthy state. This expression lets us predict, before running any experiment, which co-faults will be caught quickly and which ones need many frames. A mild joint degradation produces a small $D$ and therefore long delays. Checking the measured delays against this prediction is itself one of our tests.
Three things separate the monitor from prior work. It needs no oracle. It represents "all sensors degraded" as an explicit hypothesis that triggers a degraded driving mode rather than a forced routing choice. And its decisions come with stated error rates rather than a threshold picked by hand.
Relation to Prior Work
Two papers motivate the gap. Testing the Fault-Tolerance of Multi-Sensor Fusion Perception in Autonomous Driving Systems fuzzes Baidu Apollo with simultaneous camera and LiDAR faults and reports failure rates above 80% for LiDAR faults and above 60% for camera faults. It measures the danger but offers no runtime remedy. Resilient Sensor Fusion Under Adverse Sensor Failures via Multi-Modal Expert Fusion, which we call MoME, uses separate camera, LiDAR and fused decoders and routes each object query to the most reliable one. It was trained and tested only on isolated drops of one sensor or the other. We reuse MoME as the downstream system and supply the health signal it lacks.
Two papers give the diagnostic ideas but assume resources a moving car does not have. Interventional Root Cause Analysis of Failures in Multi-Sensor Fusion Perception Systems localizes concurrent module faults by swapping in pristine intermediate data from simulation. It reaches roughly 88% F1 on multi-fault cases, but it depends on that oracle and combines faults with a simple OR. Formalizing and Falsifying Causal Pathways of Rare Events scores explanations of rare outcomes with log-likelihood ratios, which is the same scale-free reasoning our sequential test uses. Its authors note that the conditional densities it needs are unavailable in the rare tail. Our method sidesteps this by fitting a model with six parameters to counts, rather than estimating densities in the tail.
Three papers constrain the design. Cocoon: Robust Multi-Modal Perception with Uncertainty-Aware Sensor Fusion weights sensors with conformal nonconformity scores and gains 15% mAP over static fusion under blackout. It adds about 0.5 seconds of latency per frame, and its per-class reference features become unstable for rare classes. We therefore keep the monitor at the level of object counts, not feature vectors. Tureis isolates up to five faulty smart-home sensors by reconstructing each channel from the others, masking out a channel once it is flagged, and repeating. Its authors concede that brief spikes and genuine rare events are hard to tell apart from faults. That is exactly the confusion our agreement-direction test targets. Finally, Optimal Distributed Fault-Tolerant Sensor Fusion derives Cramér–Rao-type limits for fusion when some sensors are faulty, but only for scalar estimation. We borrow its stance that fault tolerance should come with stated statistical limits.
Research Program
We will build three single-sensor detectors on nuScenes, described in nuScenes: A multimodal dataset for autonomous driving. The LiDAR detector is PointPillars, and there is also a camera-only and a radar-only detector. We match their outputs in bird's-eye view, the top-down map of the scene, within the region all three sensors cover, and bin objects by distance. Distance binning matters because small, far-away objects are missed by every sensor at once, which would break the independence the latent-class model relies on. For each bin and each sliding window we fit the error rates, compute the likelihood of each health hypothesis, and run the sequential test with a user-set false-alarm budget.
The output drives MoME's router. A sensor judged degraded has its expert masked. If every expert is judged degraded, the system raises a "no trustworthy baseline" flag instead of routing.
The analysis has three parts. First, we test how badly the monitor fails when independence is violated, using co-faults that deliberately correlate misses. Second, we compare observed detection delays with the Wald prediction. Third, we check whether real rare objects, drawn from rare nuScenes classes, are wrongly flagged as faults.
Baselines
The naive reference is static fusion with no health monitoring, meaning the single-decoder model. MoME reports that its NDS falls from 72.9 to 44.7 under a total LiDAR drop. NDS, the nuScenes Detection Score, combines mAP with errors in position, size, orientation, velocity and attributes.
The strongest existing method is MoME's own router. On single-sensor failures it reports +4.2% mAP over the previous best under total LiDAR drop and +6.7% under limited field of view, with about 4 ms of added latency. We found no reported result on Lacuna for MoME under simultaneous camera and LiDAR corruption, so we will run it ourselves. We will use the released training protocol and the same detection backbone, so that the only difference is where the health signal comes from.
Cocoon is the uncertainty-aware competitor. We will run it on the same splits with the same backbone and tuning budget, and report its latency next to ours. We will also run a Tureis-style baseline that reconstructs each modality from the others at the feature level and masks out a modality once it is flagged.
Two ablations isolate our key ingredients. The first uses the same agreement counts but makes a per-frame threshold decision with no accumulation, which isolates the value of sequential testing. The second uses two modalities instead of three, which isolates the identifiability argument.
Our monitor must meet three conditions to count as beating these baselines. It must detect co-faults with a lower delay than every baseline at the same measured false-alarm rate. It must lift MoME's NDS on co-fault splits beyond the scene-level bootstrap confidence interval. And it must keep its alarms on genuine rare objects below the stated $\alpha$.
Why not just…?
Why not just retrain MoME's router with simultaneous corruptions? This is the most serious threat, and we will run it with the same compute budget. It may improve routing when some expert is still usable. But when all three experts are degraded, the router can only choose among bad options. It has no output that means "abstain," and nothing gives its choices a stated error rate. The experiment settles the question. If co-fault-trained MoME matches our downstream NDS and our all-degraded flag adds nothing to safety metrics, the monitor is not needed.
Why not just judge each sensor's health from its own signal, such as image sharpness or LiDAR point count? Judging sensors one at a time avoids the single-fault assumption entirely, so this is a real contender and will be included as a baseline. However, it cannot see faults whose raw signal looks normal. It also falls into MoME's "calibration trap," in which a corrupted sensor outputs confident but wrong boxes. Tureis reports the same weakness for stuck readings that stay within the normal range. Cross-modal counts catch these faults because the degraded detector stops agreeing with the others. We will report results separately for faults that do and do not change a sensor's raw statistics, so that any advantage can be attributed to the right cause.
Why not just use an oracle-based root-cause analysis or conformal weighting? IRCA requires pristine replay data that does not exist on the road. Cocoon adds about 0.5 seconds of latency per frame and relies on per-class reference features that become unstable for rare classes. We will measure how Cocoon's accuracy on rare classes compares with ours, rather than simply asserting that it degrades.
Proposed Experiments
We will build co-fault splits by composing nuScenes-C environmental corruptions (fog, snow, glare) across camera and LiDAR in the common-cause pairings that FADE identified. We will add nuScenes-R hardware failures (LiDAR beam reduction, limited field of view, camera drop) on top of these. To create faults that appear mid-drive, we will inject them at random times within scenes.
The metrics fall into three groups. For diagnosis, we measure detection delay in frames at a fixed false-alarm rate per hour and the accuracy of identifying the correct fault subset. For downstream perception, we report mAP and NDS with scene-level bootstrap intervals. The intervals matter because co-fault splits hold few relevant objects, and gains of a few mAP points may not exceed sampling noise. For safety, we measure how often the all-degraded flag fires on truly unusable frames and how often it fires on genuine rare-class objects.
As an out-of-domain check, we will apply the same counting monitor to the multimodal out-of-distribution segmentation benchmarks CARLA-OOD and SemanticKITTI. There, Feature Mixing cut false positives at 95% true-positive rate (FPR@95) by about 73% on CARLA-OOD. The question is whether our monitor avoids flagging those unfamiliar objects as sensor faults.
The decisive checks are three. First, measured delays should track the Wald prediction $\log(1/\alpha)/D$. Second, the two-modality ablation should lose the ability to tell which sensor is at fault, as identifiability theory predicts. Third, the monitor should beat co-fault-trained MoME on frames where all sensors are degraded. If the third check fails, the proposal reduces to a study of how many samples such monitors need, and we will report it as that.
References
- Testing the Fault-Tolerance of Multi-sensor Fusion Perception in Autonomous Driving Systems
- Resilient Sensor Fusion Under Adverse Sensor Failures via Multi-Modal Expert Fusion
- Interventional Root Cause Analysis of Failures in Multi-Sensor Fusion Perception Systems
- Formalizing and Falsifying Causal Pathways of Rare Events
- Cocoon: Robust Multi-Modal Perception with Uncertainty-Aware Sensor Fusion
- Tureis: Transformer-based Unified Resilience for IoT Devices in Smart Homes
- Optimal Distributed Fault-Tolerant Sensor Fusion: Fundamental Limits and Efficient Algorithms
- nuScenes: A multimodal dataset for autonomous driving
- Extremely Simple Multimodal Outlier Synthesis for Out-of-Distribution Detection and Segmentation