Self-driving cars usually segment scenes by fusing a camera image with a LiDAR point cloud, which is a set of 3D points measured by a spinning laser. When such a model is moved to a car with a different LiDAR, for example from a 64-beam sensor to a 32-beam one, it degrades badly. The main reason is that the new sensor returns far fewer points at long range and on slanted surfaces. We propose a simple change to how the fusion model is trained. For each point, we sometimes delete the LiDAR feature before fusion, and the deletion probability equals how much sparser the target sensor is than the source sensor at that point's distance. The fusion layer then sees, during training, the situations it will face after deployment: points where the geometry is thin and the camera has to carry the prediction. Uniform modality dropout cannot produce this pattern, because it removes a whole sensor at random everywhere.
The bet behind this is a diagnosis. Much of the cross-sensor 3D gap may not come from bad LiDAR features. It may come from the fusion layer trusting LiDAR in places where, on the target car, LiDAR no longer deserves that trust. If the diagnosis holds, the cure belongs at the fusion layer, not in heavier geometric reconstruction. The proposal tests the diagnosis first and the method second.
Core Claim
Modality dropout, meaning training-time removal of an entire input stream, was developed to stop multimodal models from relying on one dominant input. M3 zeroes whole modality tensors in sentiment analysis and finds that skewing the drop toward the dominant modality works best. HeMIS and mmFormer use random modality subsets so that MRI segmentation degrades gracefully when sequences are missing. All three apply one drop decision per sample, and the rate is a fixed hyperparameter.
The 3D domain-shift literature shows that the gap is spatially structured. MLSP identifies prediction of local point count as its most important ingredient against density shift. Zou et al. attribute the gap to ray-casting density falloff, occlusion and sensor noise. Our claim joins these two lines. The dropout decision should be made per point, and its rate should be set from unlabeled statistics of the two sensors, so that the fusion layer learns which stream to trust as a function of local geometric support.
Concretely, let $\rho_S(d)$ and $\rho_T(d)$ be the mean local point density at range $d$ in source and target scans. Both are estimated from unlabeled scans by counting neighbors within a fixed radius. A source point at range $d$ has its 3D feature zeroed before fusion with probability
$$p_{3D}(d) = \max\Big(p_0,\; 1 - \min\big(1, \rho_T(d)/\rho_S(d)\big)\Big),$$
where $p_0$ is a small floor rate. The image feature of each point is zeroed with a fixed rate $p_{2D}$. A point never loses both streams at once. On target data during self-training, only the floor rates apply.
Relation to Existing Cross-Modal Adaptation
The closest work is camera-LiDAR unsupervised domain adaptation for 3D semantic segmentation. "Unsupervised" here means the target domain has no labels. Four methods define this space, and each differs from our proposal in a specific way.
xMUDA makes the 2D and 3D streams imitate each other's output probabilities on target data. When one stream collapses, its errors are passed to the other. Our method does not ask the streams to agree.
Mx2M masks image patches or point segments and reconstructs one modality from the other. Its masking is random, used only as a self-supervised task, and not aimed at sensor-specific sparsity.
FtD++ fuses both streams, distills the fused representation back into each stream, and down-weights pseudo-labels on which the two streams disagree. It assumes all inputs are present. It also reports that forcing LiDAR toward image features hurts geometrically ambiguous classes such as sidewalk.
CMCL aligns pixel and point features with a contrastive loss. Its own analysis flags the risk that dense image features dominate the joint space.
That last risk is what modality dropout is meant to remove. WMA constrains the design: it shows that forcing modality distributions to coincide harms performance. Our method therefore adds no alignment loss. It only changes what the fusion layer sees.
Three further papers use masking or modality switching in nearby settings. AlterMOMA masks whole camera or LiDAR branches, but only to find redundant parameters for pruning within one domain. The nighttime CMDA randomly switches between fusion inputs, but in purely 2D image-event segmentation. The detection CMDA uses the camera only as a training-time teacher and runs on LiDAR alone at test time. None of these ties the masking rate to a measured physical gap between sensors.
Research Program
The work has three stages: a diagnosis, the method, and a stratified analysis.
The diagnosis comes first because it decides whether the method is worth building. We take FtD++ trained on source only, from its released code (FtD-PlusPlus), and freeze both backbones. We then compare three ways of combining the two streams on target data. The first is the learned fusion. The second is a per-point oracle that picks whichever stream's prediction is correct; this is the ceiling for trust reallocation. The third is the 3D stream fed oracle-quality features, simulated by training it on target labels, which is the ceiling for better geometry. If the oracle-gating gain is concentrated at long range on A2D2→SemanticKITTI and is a substantial share of the total gap, the diagnosis holds. If it is negligible, the gap lies in the backbone and we stop, reporting that negative finding.
Second, we implement density-matched dropout inside FtD++'s fusion module during both source training and target self-training, keeping everything else unchanged. The density profiles are histograms over 5-meter range bins, computed once from a few hundred unlabeled scans per domain. We also test one variant for lighting shift, in which $p_{2D}$ scales with the darkness of each image patch, because Day→Night is the case where the camera rather than the LiDAR loses reliability.
Third, we stratify every result by range bin and by class. This checks two things. Gains should appear where $\rho_T/\rho_S$ is small. The sidewalk degradation that FtD++ reports should shrink, because the fusion layer can now lean on the image where geometry is ambiguous.
Baselines
The naive reference is source-only FtD++ fusion with no adaptation. Above it sits standard xMUDA. Mx2M reports xMUDA at 43.3 mIoU on A2D2→SemanticKITTI, although the xMUDA paper itself reports 48.5. We will rerun xMUDA in our pipeline and report both the cited figures and our own.
The strongest published numbers in the file come from Mx2M. Mx2M reports 53.0 mIoU on A2D2→SemanticKITTI, 67.4 on USA→Singapore and 54.6 on Day→Night. For CMCL, the file gives 66.7 on USA→Singapore and 54.5 on Day→Night. FtD++ is reported only relatively, at +9.4 mIoU over xMUDA on Day→Night; no absolute FtD++ figure was found on Lacuna. FtD++ is nonetheless our host model and the most direct competitor, because our method is an addition to it. We will rerun it with its released code, the same backbones, the same schedule and the same seeds, and our version must beat that rerun.
The fairness rule is that every variant gets the same number of training iterations and the same grid of three settings for its own key hyperparameter. "Beating" means beating the FtD++ rerun by more than seed variance, measured over three seeds, on A2D2→SemanticKITTI. Because that is the clean density-shift task, we expect the gain to concentrate in the far-range bins.
Two ablations isolate the key ingredient, the density-matched schedule. The first applies M3-style uniform per-sample modality dropout with its rate tuned. The second applies per-point dropout at the same average rate but spread uniformly across range. If either matches our method, the density coupling is not what matters.
Why not just…?
Why not just augment the source LiDAR to look like the target, by dropping beams or using a density translator? This is the most serious alternative. The density-guided translator of DGT-ST attacks exactly this gap, and its code is available (DGT-ST). Beam dropping changes what the backbone sees. Feature dropout changes what the fusion layer learns to trust. The two may be redundant, because beam-dropped inputs also produce weak 3D features at long range. We will run beam-drop augmentation alone, our dropout alone, and both together. If augmentation alone matches the combination, we will report that feature-level dropout adds nothing on sensor shift and keep it only if it still helps under full-sensor failure.
Why not just use uniform modality dropout as in M3? Uniform per-sample dropout removes LiDAR equally at 5 m and at 60 m. It therefore teaches a general fallback, not one that depends on range. The tuned-rate uniform ablation tests this directly.
Why not rely on FtD++'s disagreement-based pseudo-label filtering? That filter decides which target labels to train on. It does not change how the fusion layer weighs each stream at inference. The test is whether our method's gain persists when the filter is left on, and whether the range-stratified improvement appears even without self-training.
Doesn't Mx2M's masked cross-modal prediction already do this? Mx2M masks randomly and asks the network to reconstruct the missing content, which builds cross-modal correspondence. Our method masks according to measured sensor sparsity and asks for nothing to be reconstructed. Combining Mx2M's objective with our schedule is a natural extension, but our first comparison is against Mx2M's reported numbers.
Proposed Experiments and Decisive Checks
The main benchmarks are A2D2→SemanticKITTI, USA→Singapore and Day→Night from the xMUDA protocol, all scored by mIoU (mean intersection-over-union across classes).
We add three evaluations. The first is mIoU within range bins. The second is mIoU when the entire camera or the entire LiDAR stream is zeroed on the target domain, which tests robustness to sensor failure; none of the cited adaptation methods reports this. The third is per-class change for sidewalk, road and other flat classes.
Two outcomes would decide the question. The first is whether the oracle-gating diagnosis shows reallocable error at long range. The second is whether the density-matched schedule beats both uniform ablations and beam-drop augmentation on A2D2→SemanticKITTI, with the gain growing as $\rho_T/\rho_S$ falls. We would treat USA→Singapore, where the two LiDARs are similar, as a control. There the method should help little, and a large gain on that control would mean the mechanism is something other than density matching.
References
- M3: MultiModal Masking Applied to Sentiment Analysis
- HeMIS: Hetero-Modal Image Segmentation
- mmFormer: Multimodal Medical Transformer for Incomplete Multimodal Learning of Brain Tumor Segmentation
- Point Cloud Domain Adaptation via Masked Local 3D Structure Prediction
- Geometry-Aware Self-Training for Unsupervised Domain Adaptation on Object Point Clouds
- xMUDA: Cross-Modal Unsupervised Domain Adaptation for 3D Semantic Segmentation
- Mx2M: Masked Cross-Modality Modeling in Domain Adaptation for 3D Semantic Segmentation
- Fusion-Then-Distillation: Toward Cross-Modal Positive Distillation for Domain Adaptive 3D Semantic Segmentation
- Cross-Modal Contrastive Learning for Domain Adaptation in 3D Semantic Segmentation
- Wasserstein Modality Alignment Makes Your Multimodal Transformer More Robust
- AlterMOMA: Fusion Redundancy Pruning for Camera-LiDAR Fusion Models with Alternative Modality Masking
- CMDA: Cross-Modality Domain Adaptation for Nighttime Semantic Segmentation
- CMDA: Cross-Modal and Domain Adversarial Adaptation for LiDAR-Based 3D Object Detection
- Density-guided Translator Boosts Synthetic-to-Real Unsupervised Domain Adaptive Segmentation of 3D Point Clouds