A vehicle or robot that changes its sensors, by losing a camera, swapping a 32-beam LiDAR for a cheap 4-beam one, or adding radar, currently faces two bad options. It can keep one neural network meant to tolerate any sensor mix, which wastes compute on sensors that are absent and may be the wrong shape for the ones that remain. Or it can have engineers redesign and retrain a network for each configuration. This proposal tests a third option. We train one large "supernet," a network whose smaller sub-networks share weights so that many architectures can be extracted without retraining. We then learn a mapping from the machine-readable sensor description that a hardware abstraction layer already provides (which sensors are present, their resolution, beam count, and the compute budget) to the sub-network that should run. Choosing an architecture for a new sensor suite becomes a lookup taking seconds, not a search taking GPU-days.
The underlying scientific question is simple, and nobody in the file has answered it: does the best architecture actually depend on the sensor suite? If it does, systems with a fixed topology leave accuracy or latency unused. If it does not, one model trained to be robust to sensor dropout is enough, and per-configuration architecture design is wasted effort. Either answer is useful, and the experiments are built so they can tell the two apart.
Core Claim
Our claim is that, at a fixed latency budget, the accuracy-maximizing allocation of capacity varies systematically with sensor configuration. That allocation includes per-modality backbone width, transformer decoder depth, number of 3D object queries, and sampling points per query. We further claim the variation is predictable from the sensor descriptor alone. Formally, for a descriptor $d$ (the set of active sensors and their parameters) and a latency budget $B$, we want
$$a^*(d,B) = \arg\max_{a \in \mathcal{A}} \; \mathrm{Acc}(a; d) \quad \text{s.t.} \quad \mathrm{Lat}(a; d) \le B,$$
where $\mathcal{A}$ is the set of sub-networks of the supernet, $\mathrm{Acc}(a;d)$ is detection accuracy of sub-network $a$ when only the sensors in $d$ are present, and $\mathrm{Lat}(a;d)$ is its measured latency. What is new is the conditioning on $d$. Existing hardware-aware architecture search conditions on the compute target only, and existing sensor-agnostic models condition on nothing architectural at all.
Where This Sits
The proposal sits between three bodies of work that each cover half of the problem. On the sensor-abstraction side, Towards Sensor Data Abstraction of Autonomous Vehicle Perception Systems is a vision paper. It argues for an abstraction module that conditions on hardware metadata ("informed abstraction"), but it offers no algorithm or experiments and treats the perception network as fixed. It motivates our use of descriptors and its data-scarcity diagnosis shapes our benchmark design. AUTOFRAME shows that a hardware abstraction layer with a JSON vehicle configuration is practical at the systems level, but its perception modules are classical, hand-written code. We take its configuration file as the input our selector reads. FUTR3D supplies the method we build on. Its modality-agnostic feature sampler projects learned 3D queries into whatever sensor features exist, so one design handles camera, LiDAR, radar and downsampled-LiDAR suites on nuScenes. Its backbones, query count, and decoder depth are nonetheless hand-fixed and identical across suites, and that fixed topology is exactly what we make variable.
On the architecture-design side, MFH-NAS and MFAS search fusion topologies, but for a fixed set of input streams. MFH-NAS's discovered architectures are tied to their RGB–infrared dataset, and its search must be rerun if the sensors change. These motivate the claim that architecture matters for fusion, and they constrain us because per-configuration search is too expensive to repeat. Multi-Objective Hardware Aware Neural Architecture Search using Hardware Cost Diversity shows that searches can be conditioned cheaply on hardware cost across six edge devices, which supplies the multi-objective machinery. It assumes unimodal static inputs, however. DANA shows that training on randomly dropped sensors and varying sampling rates keeps a fixed network accurate under sensor loss in wearable activity recognition. It is the strongest argument for the "one robust model" alternative, and we reuse its training regime inside our supernet. Agentic Neural Architecture Search shows that a language model can propose architectures from data statistics but benefits from combinatorial search on top. It is a competitor for the descriptor-to-architecture step.
Research Program
The first component is a multi-setup benchmark built from existing data. Reichert et al. identify the lack of datasets recording the same scenes with different sensor suites. We partly sidestep this by deriving configurations from nuScenes (nuScenes: A multimodal dataset for autonomous driving), which has 6 cameras, a 32-beam LiDAR, and 5 radars. We remove cameras (subsets of the six), downsample LiDAR to 16, 4, and 1 beams as FUTR3D did, toggle radar, and reduce image resolution. This yields several dozen descriptors over identical scenes. We hold out a set of descriptors that are never seen during training, for example 8-beam LiDAR combined with a 3-camera front arc. This tests whether the selector extrapolates rather than memorizes. One limitation is that subsampling only degrades the suite: it does not produce new lens models or mounting positions. For those we add a CARLA test with altered camera intrinsics and placements, used only as a qualitative transfer check because the file reports no CARLA perception numbers to compare against.
The second component is the supernet. We start from FUTR3D's architecture and make five things elastic: per-modality backbone width and depth, number of object queries, sampling points per query, and decoder depth. Training samples a descriptor and a sub-network at every step, following DANA's random-dropout schedule. The loss is the usual set-prediction detection loss averaged over these samples, with the largest sub-network distilling into smaller ones as is standard for weight-shared supernets. We profile latency on one GPU and one embedded GPU board, so $\mathrm{Lat}(a;d)$ is measured rather than estimated from FLOPs.
The third component is the selector. For a set of training descriptors and budgets, we find good sub-networks by evolutionary search on the supernet, using the multi-objective procedure of MO-HDNAS. We then fit a small predictor from descriptor and budget to architecture choices. A lookup table over seen descriptors serves as the trivial version. At deployment, the configuration file is all the selector needs.
Before any of this runs on nuScenes, we run the whole pipeline on the four DANA activity-recognition datasets. This is a cheap pilot. If sensor-conditioned selection gives nothing over DANA's fixed network on low-dimensional inertial data, that is an early warning.
Baselines
The naive reference is a single FUTR3D model trained on the full sensor suite and evaluated on each degraded descriptor, with absent inputs zero-filled. This is the "do nothing" deployment. DANA's results suggest it will degrade badly: on UCI-HAR, imputation and replication baselines fell to 60% or lower when one of three sensors was removed, while DANA held about 85%.
The strongest competitor is a single FUTR3D model of the same size as our largest sub-network, trained with DANA-style random sensor dropout and resolution changes over the same descriptor distribution, the same epochs, and the same data. It is the strongest because it already achieves sensor-agnostic accuracy with no architecture change. For reference, FUTR3D reports 72.1 NDS on the nuScenes test set with the full suite. With 4-beam LiDAR plus cameras it reports 58.0 mAP in one statement and 56.8 mAP in another; the file gives both figures. Both exceed CenterPoint's 56.6 mAP with a 32-beam LiDAR, also as reported in the FUTR3D paper. We will reproduce FUTR3D on nuScenes validation under our training budget rather than compare against these test-set numbers directly. Matching FUTR3D's reported full-suite numbers within about one point is a precondition for trusting our comparison. To make the baseline fair at every latency budget, we also train width-scaled versions of the dropout-robust FUTR3D, uniformly scaled by hand to hit each budget. Beating it means higher NDS at equal measured latency, averaged over held-out descriptors, by a margin larger than the seed-to-seed spread across three seeds.
The expensive upper reference is an independent model trained per descriptor, with its own evolutionary search per descriptor in the manner of MFH-NAS. We can afford it only for four or five descriptors. It tells us how much of the per-configuration optimum our lookup recovers, and its cost in GPU-hours is the number our method must undercut. On M3FD, MFH-NAS reports 88.2% [email protected] against 85.3% for fixed-stage fusion, which suggests per-configuration search gains of a few points are realistic for the dataset it was searched on.
Two ablations isolate the key ingredient. The first, budget-only selection, chooses the best sub-network for a latency budget while ignoring the descriptor: one architecture per budget, shared across all sensor suites. The second, shuffled-descriptor selection, assigns each suite the architecture chosen for a different suite. If budget-only selection matches descriptor-conditioned selection, the sensor description carries no architectural information and our core claim fails.
Why not just…?
Why not just train one robust model with sensor dropout? This is the real threat. DANA shows that dropout training alone keeps a fixed network accurate under sensor loss, and FUTR3D already handles many suites with one design. Our answer is not to argue this away but to measure it. The budget-only ablation and the dropout-trained FUTR3D baseline are the tests. If they match our method at equal latency, the right conclusion is that architecture need not depend on the sensor suite, and we will report that. Our expectation that they will not match rests on a compute argument. A network sized for six cameras plus LiDAR spends most of its budget on camera backbones that are idle when only radar and a sparse LiDAR are present. A fixed model cannot redistribute that budget, for example into more queries or deeper LiDAR encoding.
Why not just run hardware-aware search separately for each new sensor suite? It probably finds better architectures per suite. MFH-NAS's gains over fixed fusion are evidence that per-configuration search pays off. However, it requires a fresh search, which takes GPU-days, every time a sensor changes, and it cannot respond when a sensor fails mid-operation. The per-descriptor upper reference measures exactly how much accuracy our lookup gives up in exchange for removing that search. A gap under one NDS point at a cost of seconds would justify the approach. A large gap would point toward running search offline per fleet configuration instead.
Why not just hand the configuration file to a language model and let it write the architecture? AgentNAS shows that language-model seeds are reasonable but improve substantially under search: Spherical error fell from 45.08% for the seed to 38.71% after evolution. Its candidate training also does not share weights, so it is too costly for 3D detection. We include a language-model selector that reads the descriptor and picks from the same sub-network space. This tests whether its priors about sensors beat a learned predictor, at negligible cost.
Proposed Experiments
The decisive first experiment is an architecture-ranking study on the trained supernet. For every pair of descriptors, we compute the rank correlation of sub-network accuracies at fixed latency. Correlation near one across all pairs means the best architecture does not depend on sensors and the project stops at a negative result. Low correlation between, say, camera-heavy and LiDAR-heavy descriptors means conditioning has room to help. We then report NDS and mAP on nuScenes validation for each held-out descriptor at three latency budgets, against all baselines above. We report the fraction of the per-descriptor upper reference's gain that our lookup recovers, and the wall-clock cost of selecting an architecture for a new descriptor. On the DANA pilot, the bar is DANA's reported accuracy: above 90% across sampling rates on UCI-HAR and about 85% with one of three sensors removed, at equal or lower measured latency. For the other three activity datasets, no reported result was found on Lacuna. The CARLA check asks only whether descriptor-conditioned selection degrades more gracefully than the dropout-trained baseline when intrinsics and placements change in ways nuScenes subsampling cannot produce.
References
- Towards Sensor Data Abstraction of Autonomous Vehicle Perception Systems
- AUTOFRAME - A Software-Driven Integration Framework for Automotive Systems
- FUTR3D: A Unified Sensor Fusion Framework for 3D Detection
- MFH-NAS:A Hybrid Neural Architecture Search Framework for Multimodal Fusion Object Detection
- MFAS: Multimodal Fusion Architecture Search.
- Multi-Objective Hardware Aware Neural Architecture Search using Hardware Cost Diversity
- DANA: Dimension-Adaptive Neural Architecture for Multivariate Sensor Data
- Agentic Neural Architecture Search
- nuScenes: A multimodal dataset for autonomous driving