A car or robot trained in a simulator usually carries several sensors, such as a color camera and a depth sensor. When it reaches the real world, two things go wrong at once. The simulator's images look wrong, and the simulator's depth is too clean: real depth has holes and noise that synthetic depth never had. Often the synthetic training data cannot be shipped with the model either, because it is too large or proprietary. The model therefore has to adapt using only its own weights and unlabeled real sensor streams, a setting called source-free domain adaptation. This proposal studies one design for that setting. A single network with shared weights processes every sensor. It is trained in simulation with modality dropout, meaning whole sensors are randomly blanked during training so the network learns to predict from any subset of them. Once deployed, the network adapts by making its predictions from different sensor subsets agree with one another.
The point is that modality dropout, usually treated only as a robustness regularizer, also produces several partly independent predictors inside one set of weights: color-only, depth-only, and fused. Their disagreements on real data are a label-free signal that works without source data. It tells us which pixels to trust as pseudo-labels and which sensor has drifted furthest from its simulated version. Because the weights are shared, adapting through one sensor also moves the features the other sensor uses. The deployed model keeps every sensor at inference and costs no more than a single backbone.
Core Claim
A shared-weight fusion model trained on synthetic data with modality dropout can be adapted to real data without source samples. The adaptation signal is agreement across sensor subsets rather than prediction confidence. This should beat two alternatives. The first is confidence-based test-time adaptation applied to the same model. The second is dual-branch source-free RGB-D methods, which should be matched in full-sensor accuracy at a fraction of their parameters and clearly exceeded when a sensor degrades or disappears in the real world.
Concretely, let $M$ be the set of sensors and let $p_S(x) = f_\theta(x_S)$ be the per-pixel class distribution the shared network $f_\theta$ produces from the sensor subset $S \subseteq M$ alone, with the other sensors zeroed as in training. Adaptation minimizes
$$ \mathcal{L}(\theta) = \sum_{S} \pi_S \sum_{i} m_i \, \mathrm{CE}\big(\hat{y}_i,\, p_{S,i}\big), $$
Here $\hat{y}_i$ is the pseudo-label at pixel $i$, taken from a reliability-weighted average of the subset predictions produced by a slowly updated (exponential-moving-average) copy of the network. The mask $m_i$ keeps only pixels where the subsets agree with that consensus. The term $\pi_S$ is the probability of sampling subset $S$ in a given step.
The adaptive part is the choice of $\pi_S$, which we call shift-aware dropout. The sampling probability is set proportional to each subset's current disagreement with the consensus, so the most shifted views receive the most training. Disagreement is used instead of entropy because entropy is exactly the quantity that becomes miscalibrated under simulator-to-real shift.
How This Relates to Prior Work
The motivating evidence comes from three source-free RGB-D papers. CrossMatch adapts GTA5/SYNTHIA→Cityscapes segmentation by enforcing consistency between an RGB stream and a depth-guided auxiliary stream. It then discards depth at inference. We borrow the idea that cross-modal agreement tames pseudo-label errors, but we keep depth at test time and use one set of weights instead of two streams. MISFIT fuses depth into a SegFormer through key-swapping between separate RGB and depth branches. It handles bad depth only by masking depth-invalid pixels. We replace the separate branches with a shared backbone and replace post-hoc masking with training against whole-sensor absence. Night-TTA weights RGB and thermal per pixel by prediction entropy during test-time adaptation. It uses a triple-branch student with teachers (about 79.1 ms per frame) and was tested only day-to-night on real data. Our method differs in using a single backbone, reliability from agreement rather than entropy, and a simulator-trained starting point.
The architecture comes from MAGIC. MAGIC runs all modalities as one batch through a shared backbone, with 24.73M parameters against 58.73M for CMNeXt. It learns from both robust and fragile modality features, which lifts depth-only DELIVER segmentation from 0.81% to 57.59% mIoU. MAGIC is fully supervised and has no adaptation mechanism, and its authors report weakness when all sensors degrade together. Simulator-to-real transfer is a case where both sensors are shifted at once, so it tests that weakness directly. Structured whole-modality masking is also supported by M3 and HeMIS. M3 found that modality masking beats feature dropout. HeMIS found that training on modality subsets gives graceful degradation.
Two papers constrain the design. Adversarial masking for point clouds shows that clean synthetic geometry versus holey real scans is itself a shortcut. That finding warns us that depth-only views from a simulator-trained model will be overconfident on real depth. It is the reason we gate by agreement, and it motivates adding structured hole masks to synthetic depth during source training. That paper needs source and target data together, which we cannot assume. Compositional failure in audio-visual LLMs shows that late layers can let one modality's prior dominate even when early features align. If that happens in our shared backbone, the subset views would collapse onto RGB. We would detect this as depth-only accuracy failing to improve.
Two further papers are complementary rather than competing. DIFO uses a frozen CLIP teacher but is limited to classification. OmniQuery works on 2D+3D data but requires 3% target labels.
Research Program
We would build a MAGIC-style shared segmentation backbone with lightweight per-sensor input projections and per-sensor normalization layers. Normalization is separated because statistics differ sharply between synthetic and real depth, while the attention and MLP weights stay shared.
Source training would use SYNTHIA RGB-D and SELMA with modality dropout over all non-empty subsets. The synthetic depth would also receive structured hole masks, which act as a cheap, source-side stand-in for real depth artifacts. After training, the source data would be deleted.
Target adaptation would then run the objective above on unlabeled Cityscapes RGB with stereo-derived depth, following MISFIT's depth protocol and class split. All subsets are processed in one batch pass.
We would analyze three things. The first is how subset disagreement evolves during adaptation and whether it tracks real per-sensor error measured against held-out labels. The second is whether pseudo-labels from agreement are cleaner than those from confidence at matched coverage. The third is whether adapting the shared weights transfers across sensors: after adaptation, depth-only accuracy should rise even on classes where the depth view was rarely the one being trained.
Baselines
The naive reference is the source-only shared model evaluated directly on real data. As a second reference point, MISFIT reports that standard RGB-only source-free methods on SYNTHIA→Cityscapes "typically range from 37% to 42%" mIoU.
The strongest existing method on full-sensor accuracy is CrossMatch, which reports 57.5% mIoU on SYNTHIA→Cityscapes and 57.7% on GTA5→Cityscapes. MISFIT reports 54.5% on SYNTHIA→Cityscapes and is the strongest method that actually keeps depth at inference. We would rerun both from released or reimplemented code on the same backbone family, class split, and target images, with the same adaptation iteration budget and a hyperparameter search of equal size.
Beating them has two parts. In full-sensor mode, we would need to come within about one point of CrossMatch while using roughly half the adaptation-time parameters and one backbone at inference. Under RGB degradation or loss, we would need to clearly exceed both methods; CrossMatch structurally cannot use depth at that point.
The most important baseline is the same shared, dropout-trained model adapted with entropy minimization (Tent-style) and with entropy-weighted fusion in the style of Night-TTA. This isolates agreement versus confidence as the adaptation signal. Two ablations isolate the key ingredients. The first removes modality dropout from source training, so the single-sensor views are poor and agreement should carry little information. The second unties the backbone into separate RGB and depth encoders of equal total size, which isolates the contribution of sharing. A third ablation replaces shift-aware sampling with uniform sampling.
Why not just…?
Why not just run CrossMatch and accept RGB-only deployment? On SYNTHIA→Cityscapes, CrossMatch (57.5%) already beats the RGB-D method MISFIT (54.5%). That is a real threat: on clean Cityscapes daytime images, depth may add little. The proposal's value then rests on conditions the benchmark does not stress, namely degraded or missing RGB and the cost of a single backbone. For this reason we report a corrupted-target protocol, described below, and are explicit that if full-sensor clean-target accuracy is the only criterion, CrossMatch may remain best.
Why not just train MAGIC on synthetic data and run standard test-time entropy minimization? This is the obvious two-part combination, and it is our most important baseline. The pointer to its weakness is MAGIC's own admission that it degrades when all sensors are corrupted together. The adversarial-masking paper adds that simulator-trained models are confidently wrong on real sensor artifacts. The direct evidence would be the same-model comparison of agreement versus entropy. If entropy matches agreement, the claim fails.
Why not just keep MISFIT's dual branches, which already fuse depth? MISFIT's separate encoders double the backbone. It also handles missing depth only by masking depth-invalid pixels, so it has no path for when RGB fails. The untied-backbone ablation at matched parameters tests whether sharing contributes anything beyond parameter savings.
Why not add a frozen foundation-model teacher or a few target labels? DIFO and OmniQuery show that both help. However, DIFO is classification-only and OmniQuery needs human labels. We treat both as orthogonal add-ons and exclude them from the main comparison so that it stays label-free and source-free.
Proposed Experiments and Decisive Checks
The primary benchmarks are SYNTHIA→Cityscapes and SELMA→Cityscapes in RGB-D, reporting mIoU on MISFIT's class split. For SELMA, no isolated reported number was found on Lacuna, so we would rerun MISFIT on it.
For robustness, we would evaluate the adapted models on Cityscapes validation under three RGB degradations applied by us: low light, fog, and full RGB loss. We would do the same under depth loss, and report per-subset mIoU for every sensor combination. As a second regime, we would use DELIVER, a driving-simulator benchmark with RGB, depth, LiDAR, and event sensors, as the source for multi-sensor pretraining. There the reference number is MAGIC's supervised 57.59% depth-only mIoU. On the MF RGB-thermal dataset we would run a real-to-real test-time comparison against Night-TTA's reported +13.07% mIoU over source-only, reporting latency against its 79.1 ms per frame on the same GPU.
Three outcomes would decide the question. First, agreement-based adaptation must beat entropy-based adaptation on the identical shared model; otherwise the core mechanism is not needed. Second, removing modality dropout from source training must substantially shrink that advantage; otherwise dropout is not what enables the method. Third, depth-only target mIoU must rise after adaptation in the shared model but not in the untied model; otherwise the cross-sensor transfer that justifies sharing is absent. Conversely, strong late-layer RGB dominance would appear as a flat depth-only curve, and we would report it as a negative result.
References
- CrossMatch: Source-Free Domain Adaptive Semantic Segmentation via Cross-Modal Consistency Training
- Source-Free Domain Adaptation for RGB-D Semantic Segmentation with Vision Transformers
- Test-Time Adaptation for Nighttime Color-Thermal Semantic Segmentation
- Centering the Value of Every Modality: Towards Efficient and Resilient Modality-Agnostic Semantic Segmentation
- M3: MultiModal Masking Applied to Sentiment Analysis
- HeMIS: Hetero-Modal Image Segmentation
- Adversarially Masking Synthetic to Mimic Real: Adaptive Noise Injection for Point Cloud Segmentation Adaptation
- Compositional Failure in Audio-Visual LLMs: Late-Layer Prior Dominance Under Cross-modal Conflict
- Source-Free Domain Adaptation with Frozen Multimodal Foundation Model
- Omni-Query Active Learning for Source-Free Domain Adaptive Cross-Modality 3D Semantic Segmentation