Adding a camera to a LiDAR-based 3D perception system should help, and if one sensor fails, the system should still be at least as good as the sensor that survives. In practice neither holds reliably. On driving benchmarks, LiDAR–camera fusion detectors gain only one to three points over LiDAR alone. When LiDAR is corrupted, they collapse toward zero instead of falling back to camera-only accuracy (Benchmarking the Robustness of LiDAR-Camera Fusion for 3D Object Detection). This failure has been studied for flat bird's-eye-view (BEV) detectors. It has not been studied for semantic scene completion (SSC), the task of predicting an occupancy and class label for every cell of a 3D voxel grid, including occluded cells. In SSC the two sensors are extremely unequal: LiDAR-only SCPNet reaches 36.7 mIoU on SemanticKITTI (SCPNet), while camera-only OccFormer reaches 12.32 (OccFormer). That imbalance is the textbook setting where the faster-learning modality takes over joint training and the weaker one is silenced.
We propose to build fusion so that each unimodal model sets a floor the fused model starts from and is trained not to fall below. First, train a LiDAR SSC model and a camera SSC model separately on the same voxel grid. Then connect them with sparse cross-attention that follows camera rays. Each voxel adds what the other sensor offers through a per-voxel gate that starts at exactly zero. During training, the model is also required to reproduce each expert's prediction whenever the other stream is switched off at the gate. The fused network therefore begins as the two experts and moves away from them only where cross-sensor evidence helps. It can be shut back down to whichever sensor survives.
Core Claim
A fusion model for volumetric scene completion can be both complementary and safe if it is anchored to frozen-at-init unimodal experts rather than trained jointly from scratch. Here "complementary" means it beats the best possible per-voxel choice between the two unimodal predictions. "Safe" means that under single-sensor corruption it stays at or above the surviving expert. The new elements are three. Fusion happens inside the native 3D voxel grid rather than in a flattened BEV plane. Interference is prevented by construction: zero-initialized residual gating plus expert-consistency distillation under gate-level masking, rather than input dropout or post-hoc repair. The evaluation also separates the two claims cleanly, with an "oracle late fusion" upper bound for complementarity and a fallback ratio for safety.
Relation to Prior Work
Two papers motivate the gap. The robustness benchmark (Yu et al.) built nuScenes-R/Waymo-R corruption tools and showed both the marginal-gain problem and the LiDAR dependency trap for TransFusion and MVX-Net. It proposed no fix beyond robust fine-tuning, which it called a patch. ECFusion (Eliminating Cross-modal Conflicts in BEV Space) named "signal dissolution": camera evidence for distant, sparsely scanned objects gets washed out in fusion. It repaired this in BEV with a post-hoc query-recovery module worth 2.1 mAP. We take the same failure into 3D voxels and prevent it during training instead of patching afterward.
Several papers supply the method. TransFusion (TransFusion) showed that soft cross-attention around a projected location, weighted by a Gaussian, tolerates calibration error: about a 0.5 mAP drop under a 1 m shift versus nearly 3 for hard projection. We reuse that spatial modulation, but apply it to voxel queries along camera rays rather than to BEV box queries. MonoScene (MonoScene) lifts 2D features along lines of sight (FLoSP) uniformly. Our camera expert keeps that lifting, and the fusion replaces uniform copying with learned, per-voxel attention. Gated cross-attention (Kim et al.) supplies the residual gate form. MedMIX (MedMIX) shows that sample-specific modality weighting degrades gracefully under missing inputs, with AUROC falling only from 0.7119 to 0.6914 as missingness rises from 0.1 to 1.0. That result concerns non-spatial medical diagnosis. We need the gate per voxel, because sensor reliability varies with range and occlusion within a single scene.
Other work constrains the design. 3DMV (3DMV) and Two Stream SSC (Two Stream 3D Semantic Scene Completion) show that fusing learned 2D features at the middle of the network gives large gains: 54.4% to 70.1% on ScanNet from a single RGB view. That argues against pure late fusion. Their concatenation, however, has no protection when a view is wrong. OccFormer shows that collapsing height loses overhanging structure, which rules out a BEV fusion shortcut for this task. SCPNet shows that downsampling erases thin objects, costing 4.1 mIoU, so our attention must run at full voxel resolution on a sparse set rather than on a coarsened grid. Rank-targeted fusion (Countering Multi-modal Representation Collapse) detects modality collapse by effective rank in 2D video models. We use effective rank only as a diagnostic, not as the mechanism. It is a competing remedy we compare against.
Research Program
The LiDAR expert is an SCPNet-style multi-path 3D convolutional network. The camera expert is a MonoScene/OccFormer-style lifted volume on the same grid. Both are trained to convergence alone.
Fusion layers sit at two depths of the decoder. At each fusion layer, a voxel $v$ with LiDAR feature $h_v^L$ forms a query. It attends to the $K$ image tokens nearest its projected pixel, weighted by a Gaussian whose width is learned, which buys tolerance to calibration drift. Symmetrically, the camera-branch voxel attends to LiDAR voxel features in a small 3D neighborhood. To keep cost linear in voxel count, queries are restricted to voxels the experts mark as occupied or near the occupied frontier. The update for the LiDAR stream is
$$\tilde h_v^L = h_v^L + g_v \cdot \mathrm{CA}(h_v^L, \{f_u\}_{u \in \mathcal{N}(v)}),$$
where $f_u$ are image tokens in the projected neighborhood $\mathcal{N}(v)$ and $\mathrm{CA}$ is the cross-attention defined above. The gate $g_v \in [0,1]$ is predicted from both features and from cheap reliability cues: LiDAR return density around $v$, and the entropy of the camera expert's prediction. The camera stream gets the mirror-image update, and a light head merges the two streams. The final gate layer is initialized so that $g_v = 0$. At step zero the network is therefore exactly the two experts.
Training adds one term to the usual SSC losses. With probability $p$, one stream's gates are forced to zero. The output is then distilled toward the frozen prediction of the surviving expert:
$$\mathcal{L} = \mathcal{L}_{\text{SSC}} + \lambda\, \mathbb{E}_{m}\big[\mathrm{KL}(\text{expert}_m \,\|\, \text{fused}_{\setminus \bar m})\big],$$
where $m$ ranges over the two modalities, $\text{fused}_{\setminus \bar m}$ is the fused model's prediction with the other modality's gates zeroed, and $\lambda$ weights the term. At test time, gates also respond to corruption without any explicit masking. We test whether that learned response matches the masked behavior.
The analysis program tracks three quantities through training. We record per-modality gradient norms through the fusion layers, the effective rank of fused voxel features, and gate maps as a function of range and class. These test the mechanistic story that zero-initialized anchoring prevents the early takeover described in analyses of modality competition (Huang et al.).
Baselines
The naive reference is the LiDAR expert alone. SCPNet reports 36.7 mIoU on the SemanticKITTI test set (SCPNet), and we retrain it ourselves under our budget for the validation comparisons. The camera expert alone (OccFormer reports 12.32 mIoU; MonoScene 11.08) sets the other floor.
The strongest fusion competitor is jointly trained mid-level fusion. We build it in two variants. One uses 3DMV-style backprojection with concatenation into the LiDAR network. The other applies TransFusion-style Gaussian-modulated cross-attention to the same voxel queries, without gating, anchoring, or distillation. This second variant is the closest existing design carried into 3D. No reported result on Lacuna was found for multimodal LiDAR–camera SSC on SemanticKITTI, so these baselines must be run by us. All fusion variants share backbones, grid, data, schedule, and a matched tuning budget of the same number of hyperparameter trials. Each is reported with and without modality dropout (ModDrop) and with and without corruption augmentation from the robustness toolkit.
We also include RTF channel blending (Countering Multi-modal Representation Collapse) inserted at the same fusion points. In addition, we run confidence-weighted per-voxel late fusion of the two experts, plus its oracle version, which picks the correct expert per voxel using ground truth.
Ablations remove one ingredient at a time. We test joint training from scratch instead of expert anchoring, nonzero gate initialization, plain modality dropout instead of gate-level masked distillation, global instead of per-voxel gates, and dense downsampled attention instead of sparse full-resolution attention.
Beating the baselines means two things. On clean data, the fused model must exceed both the jointly trained cross-attention baseline and confidence-weighted late fusion in mIoU. Under every single-sensor corruption, its fallback ratio (fused mIoU divided by surviving-expert mIoU) must stay at or above 1.0, where the joint baselines are expected to fall below it.
Why not just…?
Why not just train joint fusion with modality dropout? This is the real threat for the safety claim. Dropout at the input forces each pathway to work alone. The robustness benchmark found that noise augmentation only patched models without curing camera suppression (Yu et al.), but that evidence comes from BEV detection, not SSC. Our ablation replaces our masked distillation with standard dropout, keeping everything else fixed. If dropout matches both fallback ratio and clean mIoU, we will report that the anchoring is unnecessary. The paper's contribution then reduces to the volumetric cross-attention and the evaluation protocol.
Why not just ensemble the two experts per voxel? Late fusion guarantees a floor for free, so it is the hardest competitor on robustness. Its limit is that it can only choose between predictions that already exist. It cannot complete an occluded far-range voxel that neither expert labels correctly, which is where 3DMV-style feature-level gains come from. The oracle late-fusion bound makes this testable: if our model does not exceed the oracle selector's mIoU, there is no evidence of genuine feature-level complementarity, and the simpler ensemble should be preferred.
Why not just fuse in BEV, where TransFusion and ECFusion already work? SSC labels every height cell. OccFormer documents that collapsing height loses overhanging and multi-level structure, and expanding a BEV map back to voxels is heuristic. We will still run a BEV-fusion-then-lift variant. We will report per-class results for classes with vertical structure, such as poles, trunks, and vegetation, where the difference should appear if it exists.
Why not just use rank-targeted blending? RTF reported an average gain of up to 3.74% accuracy against modality collapse in RGB-D video. It blends channels globally, with no notion of where in space a sensor is reliable. We include it at the same insertion points, so the comparison directly tests whether spatial, per-voxel gating matters.
Proposed Experiments
The primary benchmark is SemanticKITTI SSC, using LiDAR plus the front camera and reporting validation mIoU and IoU. Results are broken down by range bin and by small-object classes, where signal dissolution is expected. Robustness is evaluated by porting the robustness-benchmark corruptions to this setting: LiDAR field-of-view limits, object failure with no returns, camera lens occlusion, extrinsic shifts up to 1 m, and stuck frames. For each corruption we report the fallback ratio relative to the surviving expert. NYUv2 SSC from RGB-D serves as an indoor check. There, Two Stream reports 34.1% IoU on NYU Kinect and MonoScene 26.94 mIoU from RGB alone, which tests whether anchoring helps when depth comes from a different sensor.
Three decisive checks settle the claims. First, on clean data, does the fused model beat oracle late fusion? This tests complementarity. Second, under total LiDAR loss, does it stay at or above camera-only mIoU while the joint cross-attention baseline does not? This tests safety. Third, do per-modality gradient norms and fused effective rank show the camera pathway staying active under anchoring and collapsing under joint training? This tests the mechanism.
The proposal fails cleanly in two cases: if the modality-dropout ablation closes both gaps, or if late fusion matches the clean mIoU. Each outcome would be reported as a finding about what volumetric fusion actually needs.
References
- Benchmarking the Robustness of LiDAR-Camera Fusion for 3D Object Detection
- SCPNet: Semantic Scene Completion on Point Cloud
- OccFormer: Dual-path Transformer for Vision-based 3D Semantic Occupancy Prediction
- Eliminating Cross-modal Conflicts in BEV Space for LiDAR-Camera 3D Object Detection
- TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers
- MonoScene: Monocular 3D Semantic Scene Completion
- An Interpretable Framework for Drug-Target Interaction with Gated Cross Attention
- MedMIX: Modality-Internal Expert Fusion for Multimodal Medical Diagnosis
- 3DMV: Joint 3D-Multi-view Prediction for 3D Semantic Scene Segmentation
- Two Stream 3D Semantic Scene Completion
- Countering Multi-modal Representation Collapse through Rank-targeted Fusion
- ModDrop: adaptive multi-modal gesture recognition