Self-driving detectors that combine a laser scanner (LiDAR) with cameras tend to lean almost entirely on the laser. When the LiDAR is degraded, these models collapse instead of falling back to what the cameras can see. This proposal argues that the leaning is not uniform across the outputs a detector must produce, and that orientation is where it does the most avoidable damage. A sparse point cloud often pins down the axis an object lies along, but it cannot always tell which end is the front. A camera image, with its headlights, taillights and windshield, often settles exactly that question. Current fusion detectors predict orientation as one continuous number. A single number cannot express "this axis, direction unknown," so there is no place in the output where the camera's specific contribution can be measured, supervised, or credited.
We propose to split orientation into a discrete distribution over axis bins plus a front/back choice. The approach rests on discrete orientation classification, meaning predicting a probability over angle bins instead of regressing an angle. Each sensor gets its own orientation head, and the fused orientation is the product of the per-sensor distributions. On top of this split head we apply the balancing methods developed for modality dominance (the tendency of multimodal networks to learn mostly from the easiest input). These methods have so far been applied to one global classification loss. Here they would act only on orientation, leaving box position and size training untouched. The bet is that this targeted credit assignment yields camera-supported orientation that survives LiDAR failure. It should do so without the cost to localization that global rebalancing would impose.
Core Claim
Modality dominance in 3D detection depends on the sub-task, and its cost on orientation falls mostly on heading direction. A factored discrete orientation head turns that into something measurable and fixable. The factors are an axis distribution over 180° and a front/back bit, with one head per sensor, multiplied together. The claim is that this head, trained with per-sensor auxiliary losses and orientation-scoped gradient balancing, will close most of the gap on orientation error under LiDAR corruption. The gap in question is between a fusion model and a camera-only model, and the improvement should not cost clean-data accuracy.
What is new is the pairing of a representation that separates what each sensor can know about orientation with balancing that is applied at the granularity of that representation. Neither literature does this. The discrete-orientation work is unimodal throughout. Circular Smooth Label classifies aerial-box angles over 180 bins with a smoothing window. Dense Label Encoding compresses those bins into Gray-coded bits. MultiBin combines bin confidences with residuals from a single camera crop. The balancing work assumes one scalar classification loss. That includes G-Blend, which reweights unimodal and fused heads by an overfitting-to-generalization ratio. It includes OGM/OPM, which damps the dominant modality's gradients or features. It also includes MMPareto, which resolves conflicts between fused and unimodal gradients and rescales them to keep gradient noise.
Motivation and Constraints from Prior Work
Two results motivate the proposal. Benchmarking the Robustness of LiDAR-Camera Fusion built nuScenes-R and Waymo-R corruptions and found that fusion models such as TransFusion fall to near zero when LiDAR fails, rather than to camera-only levels. That benchmark reports only system-level mAP and NDS and treats the orientation head as opaque. MultiBin shows that a camera alone can get orientation right, with roughly 3° error on easy KITTI cars. Together these suggest that orientation is where camera evidence is most wasted. Broader reports of fragile fused representations (Luo et al., Ramaneswaran S et al.) point the same way.
Two architectures set the design space. TransFusion initializes object queries from LiDAR bird's-eye-view (BEV) features and only then attends to images, which builds LiDAR-first behavior into the architecture. BEVFusion puts both sensors into a shared top-down grid, concatenates them, and inherits a regression head from CenterPoint that predicts sin and cos of the heading. We build on BEVFusion because its fusion is a plain concatenation, which makes per-sensor branches easy to attach.
Three results constrain the design. MMPareto shows that off-the-shelf Pareto multi-task optimization does worse than uniform weighting on multimodal problems, so we use MMPareto's noise-preserving variant. MMPareto's deliberate gradient noise sits uneasily with Gray-coded bits, where noise on a high-order bit means a 90° or 180° jump. For that reason we use CSL-style smoothed bins for the axis and a single explicit direction bit, not DCL codes. Finally, G-Blend reports that plain dropout does not fix multimodal underperformance on Kinetics. That evidence comes from action classification, not detection, so we treat it as suggestive and test modality dropout directly.
Research Program
The work has a diagnostic stage, a method stage, and a robustness stage.
The diagnostic stage asks whether the premise is true before any building. We take a trained BEVFusion and decompose its nuScenes orientation errors into axis error (angle error modulo 180°) and heading flips (error above 135°). We do this separately on clean data, with the camera branch zeroed, and with LiDAR corrupted using the nuScenes-R toolkit. We also measure per-branch gradient norms flowing from the orientation loss and from the translation loss during training. The goal is to see whether the camera receives disproportionately little orientation gradient early on, as the dominance literature would predict.
The method stage replaces the CenterPoint heading head with a factored head. It has $N=180$ axis bins on $[0^\circ,180^\circ)$ with a circular smoothing window, as in CSL. For classes that are symmetric in practice, such as barriers and cones, the window period is adapted per class, in the spirit of DCL's aspect-ratio-sensitive period. A binary front/back logit and a small continuous residual complete the head. Three copies of this head read the LiDAR BEV map alone, the camera BEV map alone, and the fused map. The fused orientation distribution is
$$p(\theta)\propto \exp\big(z_L(\theta)+z_C(\theta)+z_F(\theta)\big),$$
where $z_L$, $z_C$ and $z_F$ are the logits over joint (axis, direction) states from the LiDAR, camera and fused heads. Adding logits means a sensor whose distribution is flat contributes nothing. This creates a natural fallback path if a corrupted LiDAR yields flat axis logits. Whether it does so after ordinary training is an empirical question addressed by the robustness stage. The loss is applied to the fused product and to each unimodal head. MMPareto's conflict rule is then applied, but only to the orientation parameters and the branch features they read: sum when gradients agree, take the Pareto direction and rescale when they conflict. Translation, size and velocity regression train with the standard loss.
The robustness stage tests whether sensor-specific orientation competence survives into the fused model under every nuScenes-R LiDAR and camera corruption. It also checks whether the camera head's front/back accuracy inside the fusion model matches a camera-only model trained alone, which mirrors MMPareto's finding that its audio branch beat a standalone audio model.
Baselines
The naive reference point is BEVFusion as published, with its CenterPoint sin/cos regression head and uniform joint training. We retrain it under our exact schedule. Its reported clean figure is 70.2% mAP on nuScenes (BEVFusion), and our retrained version must come close to that before any comparison counts. TransFusion is the strongest LiDAR-first fusion competitor, with 68.9% mAP and 71.7% NDS on the nuScenes leaderboard (TransFusion). We searched for mean orientation error (mAOE) for either model, and for any nuScenes-R orientation breakdown, and found no reported result on Lacuna; we will measure both ourselves.
The strongest competitor on the actual question is BEVFusion trained with modality dropout plus corruption-style augmentation. This is the "robust fine-tuning" family evaluated in the nuScenes-R benchmark, and it attacks LiDAR dependence directly with no change to the head. A second strong competitor is global MMPareto applied to the whole detector, with unimodal auxiliary detection heads. This tests whether scoping balancing to orientation matters. MMPareto's reported 75.13% on CREMA-D, against 71.10% for uniform training and 72.01% for G-Blend, is used only to check that our implementation reproduces it before it is moved to detection. We also include G-Blend and OGM run the same global way.
Three ablations isolate the key ingredients. The first is the factored discrete head with uniform training and no per-sensor heads, which tests whether the representation alone helps. The second is per-sensor heads with a single unfactored 360-bin distribution, which tests whether separating axis from direction matters. The third replaces the product of experts with fused-feature-only prediction while keeping the auxiliary losses.
All runs share data, epochs, backbone checkpoints and a fixed hyperparameter search budget of the same number of trials per method. Results are averaged over three seeds. Beating the field means two things together. On clean nuScenes, NDS must be no worse than the strongest baseline. Under LiDAR corruption, the fallback gap must shrink by at least half relative to modality dropout; the fallback gap is the difference in mAOE between the fused model and the separately trained camera-only model. Clean mAP must stay within half a point of the plain model.
Why not just…?
Why not just keep sin/cos regression and add modality dropout? This is the real threat, and it may well produce most of the robustness gain. The case against it as a complete answer is narrower than "dropout fails." Dropout teaches the fused head to cope without LiDAR, but it does not tell the model which sensor to trust for which part of orientation when both are present and partly corrupted. It also cannot represent a two-peaked heading distribution, the situation where the axis is known and the direction is not. The evidence that dropout is insufficient comes from classification (G-Blend on Kinetics) and from the nuScenes-R benchmark's limited fine-tuning, neither of which used orientation-level metrics. We run dropout as a first-class baseline and also stack it on our method. If dropout alone matches our fallback gap and flip rate, the central claim fails and we will report that.
Why not just swap in CSL or MultiBin bins and leave training alone? The factored head with uniform training is our first ablation, precisely because the representation by itself might be the whole story. The discrete-orientation literature shows bins beat regression in single-sensor settings, for example the roughly 4% AP75 gain on OHD-SJTU for CSL. Nothing shows that bins alone stop the camera branch from being starved of gradient while LiDAR orientation loss drops quickly. The diagnostic gradient-norm measurements and this ablation together will show whether rebalancing adds anything beyond the change of head.
Why not just run a camera-only detector alongside and ensemble its heading? This is a cheap and honest alternative, and we include a late-fusion ensemble: the BEVFusion output with headings replaced or averaged by a camera-only model when LiDAR point counts are low. It doubles inference cost and needs a hand-set switching rule. It also cannot use camera evidence when LiDAR is present but ambiguous about direction. The product-of-experts head should match it under hard LiDAR failure and beat it on heading flips in clean but sparse long-range cases. If it does not, the ensemble is the better engineering answer.
Why not just apply MMPareto or G-Blend globally? The global variants are baselines. The concern is that translation regression is exactly where LiDAR should dominate, so damping LiDAR gradients everywhere could degrade localization in order to help orientation. We will report mean translation error (mATE) alongside mAOE to measure that trade-off directly.
Proposed Experiments and Decisive Checks
The primary benchmark is nuScenes with its official orientation, translation and composite scores (mAOE, mATE, mAP, NDS), together with the nuScenes-R LiDAR and camera corruptions. Waymo-R is the transfer check. KITTI orientation similarity (AOS), the KITTI orientation score, is a secondary check against MultiBin-style camera orientation. Beyond the official metrics, we report heading-flip rate, axis error, and the fallback gap, broken down by distance band and LiDAR point count.
The first decisive check is the diagnostic. If heading flips account for only a small share of BEVFusion's orientation error, and the error stays small under LiDAR corruption, then the factored design targets a non-problem. In that case we would redirect the work toward axis error under corruption, or abandon it. The second check is the uniform-training ablation against the full method: if the two are indistinguishable, the contribution reduces to "use binned orientation in fusion." The third check is modality dropout against the full method under nuScenes-R. A smaller fallback gap and flip rate at equal clean NDS is the result that would justify the proposal. Finally, we test whether the camera head inside the fused model reaches the front/back accuracy of the standalone camera model. If it does, the balancing is doing what the dominance literature claims, at the level of a single geometric quantity.
References
- Arbitrary-Oriented Object Detection with Circular Smooth Label
- Dense Label Encoding for Boundary Discontinuity Free Rotation Detection
- 3D Bounding Box Estimation Using Deep Learning and Geometry
- What Makes Training Multi-Modal Classification Networks Hard?
- On-the-Fly Modulation for Balanced Multimodal Learning
- MMPareto: Boosting Multimodal Learning with Innocent Unimodal Assistance
- Benchmarking the Robustness of LiDAR-Camera Fusion for 3D Object Detection
- Multi-Grained Multimodal Interaction Network for Entity Linking
- MMER: Multimodal Multi-task Learning for Speech Emotion Recognition
- TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers
- BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird’s-Eye View Representation