Teaching Fusion Residuals What to Do When One Sensor Shifts

When a model sees two sensors at once, such as a color camera and a depth camera, one sensible way to combine them is to treat the second sensor as a correction. The network keeps the camera features and adds a learned "residual" that holds only what the camera is missing. This residual fusion works well inside a single dataset. Nobody has tested whether it survives domain generalization, the setting where a model trained on some environments or devices must work on an unseen one with no access to its data. Under that kind of shift, what the camera is missing changes. A new lighting regime or a new device can make the color stream unreliable in places where it was fine during training. A different depth sensor can make the depth stream noisy or full of holes. A residual learned only on clean source data has never seen either situation. It has no reason to grow when the camera fails, and no reason to shrink when the depth goes bad.

We propose to train the residual on counterfactual pairs to handle both cases. Within one training example, we perturb only the color stream in a domain-like way and require the fused prediction to stay correct, which forces the residual to carry more. Separately, we corrupt only the depth stream and require the fused prediction to fall back gracefully toward the color-only answer, which forces the residual to filter. We study this on dense indoor and driving-scene RGB-D semantic segmentation under strict domain generalization, a combination the literature has not evaluated. The central measurable claim is that, on an unseen domain, the per-pixel size of the residual should predict where the color-only model is wrong.

Core Claim

Residual fusion writes the fused feature as $Y = X_A + g \odot \mathcal{R}(S(X_A, X_B))$. In this expression, $X_A$ is the primary (RGB) feature map, $X_B$ is the secondary (depth) feature map, $S$ is a 1×1-convolution selection module that filters each stream before merging, $\mathcal{R}$ is the residual branch, and $g \in [0,1]$ is a per-pixel gate computed from both streams. This is the form used by Progressively Complementarity-Aware Fusion Network for RGB-D Salient Object Detection, with the select-then-fuse step taken from Deep feature selection-and-fusion for RGB-D semantic segmentation. Both papers show the structure beats concatenation in-domain. Our claim is that the structure helps under domain shift only if the residual's magnitude is trained to follow the reliability of each stream. That training requires one-sided perturbations: perturb one sensor and leave the other untouched. Symmetric augmentation does not do this, and neither does dropping a whole modality.

What is new is the training signal and the evaluation regime, not the residual block itself. The three closest existing methods attack the problem with whole-modality removal or alignment:

Dropout tells a model "this sensor may vanish." It does not tell the model "this sensor is present but has drifted in these pixels." The second case is what a new device or new lighting produces. Alignment-based methods also sit in tension with SimMMDG's own "alignment trap" result: forcing modalities to agree throws away the private cues that matter when one stream degrades. A gated residual sidesteps that tension. The private depth information is never forced into agreement with RGB. It is added only where it is needed.

Some prior work motivates this proposal. Residual Cross-Modal Fusion Networks for Audio-Visual Navigation found that learned reliance on each modality shifts with scene complexity. Its fusion relies on fixed learned scales and has no domain generalization objective, which is the gap we target. Other work constrains the design. Cross-modal Representation Flattening shows that plain sharpness-aware minimization fails in multimodal settings, so we do not rely on flatness alone. The Domain Generalization for Semantic Segmentation survey warns that foundation-model backbones partly bypass domain generalization through proxy data, so we must control for the backbone.

Research Program

The model is a SegFormer-style or ResNet-101 encoder-decoder with two encoder streams and the gated residual fusion above at every scale. A lightweight auxiliary head reads $X_A$ alone and produces an RGB-only segmentation. That head serves two purposes: it provides a fallback target, and it gives us a diagnostic.

Each training image gets two one-sided views. The first is an RGB-shifted view $\tilde x_A$, with depth left unchanged. We build it by swapping Fourier amplitude spectra across source domains (the input-level style transfer that MISFIT uses for adaptation, used here only among source domains), plus random local photometric corruptions in patches. The second is a depth-corrupted view $\tilde x_B$, with RGB left unchanged. We build it from sensor-style depth damage: holes, quantization at different depth ranges, and edge bleeding.

The training objective is

$$\mathcal{L} = \ell(Y, y) + \ell(\tilde Y_A, y) + \lambda_1\,\mathrm{KL}\big(p(\tilde Y_B)\,\|\,\mathrm{sg}[p(Y) \text{ if depth helped, else } p_A]\big) + \lambda_2\, \mathbb{1}[\hat y_A = y]\,\|g \odot \mathcal{R}\|_1 .$$

The terms are defined as follows:

  • $\ell$ is per-pixel cross-entropy against labels $y$.
  • $Y$, $\tilde Y_A$ and $\tilde Y_B$ are the fused outputs on the clean, RGB-shifted and depth-corrupted inputs.
  • $p(\cdot)$ is the softmax prediction, $p_A$ is the RGB-only head's prediction, and $\hat y_A$ is its label.
  • $\mathrm{sg}$ stops gradients.
  • The indicator restricts the sparsity penalty to pixels where RGB alone was already right.

The second term forces the residual to compensate when RGB drifts. The third forces the fused output under bad depth to stay near the best clean answer instead of being corrupted by the noise. The fourth keeps the residual quiet where it adds nothing, which makes its magnitude interpretable.

The analysis side tests whether the residual behaves as claimed on held-out domains. We measure how well per-pixel $\|g \odot \mathcal{R}\|$ ranks RGB-only error pixels, reported as the area under the ROC curve. We also measure whether fused accuracy ever drops below RGB-only accuracy, which is the "negative fusion" failure that MBCD reports in action recognition.

Baselines

The naive reference points are two networks with the same backbone and decoder, trained with plain empirical risk minimization on pooled source domains: one fuses by concatenation and one by element-wise addition. We also train an RGB-only model, because if depth does not help on unseen domains, the whole question is moot.

The strongest in-domain fusion competitor is the selection-and-fusion network of Deep feature selection-and-fusion for RGB-D semantic segmentation, which reports 52.0% mIoU on NYUDv2 and 50.6% on SUN RGB-D in-domain. Its selection modules make it the closest architecture to ours without our training signal, so it isolates the training signal directly. We re-implement it on our backbone and train it under empirical risk minimization on our domain splits.

The strongest domain generalization competitors are ports of methods built for classification:

  • VGDP-style fixed 20% modality dropout on our residual network.
  • MBCD's adaptive modality dropout plus gradient consistency plus distillation from an averaged teacher.
  • SimMMDG's shared/private split with its distance loss, applied to per-pixel features.

Their reported numbers are on action recognition only. On HAC, SimMMDG reports 71.10% mean accuracy and representation flattening reports 71.91%. On EPIC-Kitchens, MBCD reports 67.58% against an empirical risk minimization baseline of 50.55%. We found no reported result on Lacuna for any of these methods on dense RGB-D segmentation under domain generalization.

On the driving benchmark, the relevant reported numbers are for different settings. MISFIT reaches 54.5% mIoU on SYNTHIA→Cityscapes with RGB-D, but it uses unlabeled target data. The survey reports 40–48% mIoU for classic ResNet-101 RGB-only domain generalization, and 68.3% (SoRA) and above 71% (MFuser) with foundation backbones. We treat MISFIT as a soft ceiling for a method that sees the target domain. We treat the foundation-model numbers as a separate track, run only with matched backbones.

Every method gets the same backbone and initialization, the same source data and augmentation pool, the same number of iterations, and the same hyperparameter budget of 12 trials. All model selection uses a held-out source domain, never the target.

Beating the baselines means two things. The method must exceed the best ported domain generalization method by more than the seed-to-seed standard deviation, averaged over held-out domains. It must also never fall below RGB-only on any held-out domain.

The ablations isolate the key ingredient:

  • Residual network with no one-sided views.
  • Residual network with only the RGB-shifted view.
  • Residual network with only the depth-corrupted view.
  • Residual network without the sparsity term.
  • Concatenation network trained with both one-sided views. This is the decisive ablation explained below.

Why not just…?

Why not just apply the same one-sided augmentation to a concatenation network? This is the real threat. If the training signal does all the work, the residual structure is decoration. The concatenation-plus-one-sided-views ablation settles it directly. The proposal predicts two differences. First, a smaller accuracy gap in favor of the residual model on held-out domains. Second, and more decisive, a large difference in the per-pixel diagnostic: a concatenation network has no separable correction term whose size can be read off. If the accuracies match and only interpretability differs, we will report the contribution as a training recipe plus a reliability signal, not as an architecture.

Why not just use modality dropout, fixed or adaptive? Dropout teaches robustness to a sensor disappearing, but cross-device and lighting shifts leave the sensor present and partly wrong. VGDP and MBCD were validated on whole-input policies and clip-level labels, where all-or-nothing removal is a reasonable proxy. Dense segmentation needs pixel-level decisions about which stream to trust. Both dropout variants are baselines run on our residual network, so any gain is attributable to one-sided partial perturbation rather than to the architecture.

Why not just use an RGB-only model with a strong pretrained backbone? The survey's 68.3% and above-71% mIoU figures for foundation-model RGB-only domain generalization on Cityscapes are a serious argument that depth is unnecessary. We answer it two ways. We run every method with both a classic and a foundation backbone. We also report results on SUN RGB-D's cross-device split, where the shift lies largely in the depth sensor and in geometry, which large RGB pretraining does not cover. If depth fusion adds nothing over a strong RGB-only model on both, that is a publishable negative result.

Why not just apply sharpness-aware minimization to the whole fused model? Cross-modal Representation Flattening reports that this fails in multimodal training (64.72% on HAC versus 71.91% for representation flattening) because the faster-learning modality dominates. We include it as a cheap baseline, but the evidence already argues against expecting it to suffice.

Proposed Experiments

The primary benchmark is SUN RGB-D (SUN RGB-D benchmark), which aggregates images from several capture devices. We use the capture device as the domain label and run leave-one-device-out training, reporting mIoU on each held-out device and their average. NYUDv2 serves as an additional held-out target when it is excluded from training.

The second benchmark is synthetic-to-real driving: SYNTHIA→Cityscapes and SELMA→Cityscapes with depth, following MISFIT's data setup but with no target images during training. This tests whether the effect carries over from indoor scenes to outdoor ones. As a transfer check outside dense prediction, we apply the training signal with RGB video as primary and audio and optical flow as secondary on HAC. There the reported 71.91% (representation flattening) and 71.10% (SimMMDG) are the numbers to compare against.

Three checks decide whether the idea is right.

The first is the reliability diagnostic. On held-out domains, the ROC area of residual magnitude for predicting RGB-only error pixels should clearly exceed 0.5 for our model. It should exceed that of the residual model trained without one-sided views, which we expect to sit near chance on unseen devices.

The second is stress testing under controlled corruption. On held-out domains, we inject increasing depth damage at test time and plot mIoU against severity. Our model should degrade toward the RGB-only curve and never cross below it. The concatenation, dropout and SimMMDG-style baselines are expected to cross below. Repeating the test with RGB lighting and style damage, our model should stay above both the RGB-only curve and the concatenation-plus-one-sided-views curve.

The third is the decisive ablation itself. Comparing concatenation plus one-sided views against residual plus one-sided views, at equal training budget, tells us whether the residual form is necessary or whether the training signal alone carries the result. We will state that answer plainly, whichever way it falls.