Aligning Only the Global Context: Scale-Selective Adversarial Fusion for RGB-Thermal Segmentation

When a network fuses a color camera with a thermal camera, the two feature streams look statistically different even when they show the same street. A common fix is adversarial modality alignment. A small classifier, the modality discriminator, tries to tell which sensor a feature came from, and the feature extractors are trained to fool it, usually through a gradient reversal layer that flips the sign of the discriminator's gradient. Applied everywhere, this fix removes too much. Thermal images have almost no fine texture and carry temperature contrast that RGB lacks at night. A discriminator looking at small local patches will always find those differences, and forcing them away erases exactly what makes thermal useful. What should be the same across sensors is the scene layout: where the road, cars and people are, and how they relate across the image. That layout is only visible to features with a large receptive field, meaning the region of the input a feature can "see."

This note proposes to decide where alignment is applied by receptive field scale. The discriminator reads features that have passed through criss-cross attention, a cheap global-context module, while local texture features are fused without any invariance pressure. The discriminator is first trained alone on frozen features. That warm-up does two jobs. It avoids the early noisy gradients that damage pretrained backbones. It also works as a measuring instrument: its accuracy at each scale tells us how much modality-specific information lives there before we commit to removing any of it.

Core Claim

In registered multi-sensor dense prediction, adversarial invariance helps when it is imposed on globally routed context features and hurts when imposed on local features. A modality discriminator pre-trained on frozen features makes it safe to put the reversed gradient after a global attention block, where an untrained discriminator's noise would otherwise be spread across every pixel at once. We expect three measurable results on MFNet and PST900. First, frozen-feature discriminators identify the modality almost perfectly at local scale and noticeably less well after global routing. Second, aligning only the global stage beats aligning all stages, with the gap concentrated in night scenes and thermally distinctive classes such as pedestrians. Third, discriminator warm-up gives lower seed variance and less drift of the pretrained features than both unstaged training and a smoothly ramped reversal weight.

Relation to Prior Work

The pieces exist separately, and the file shows no prior work combining them.

Adversarial Discriminative Heterogeneous Face Recognition is the closest work. It aligns near-infrared and visible faces with an adversarial feature loss plus a class-wise variance term, and it treats scale through a hand-cropped periocular local path alongside a global path. Its local path is chosen by hand, adversarial alignment is applied to the final embedding, and the discriminator is not staged. We instead learn global context with attention, and we choose which scale to align by measuring modality identifiability at each scale.

Two papers motivate the invariance pressure and one warns against overdoing it. A Closer Look at Multimodal Representation Collapse shows that joint training tends to suppress the weaker modality and uses per-branch modality discriminators to counter this. It was tested only on clinical and digit data, trained without staging, and its authors flag hyperparameter sensitivity. They also assume modalities carry comparable label information, which RGB and thermal at night do not. X3KD aligns camera and LiDAR bird's-eye-view tokens adversarially, concurrently with training. The warning comes from Chen et al., who note that aggressive alignment discards task-critical sensor-specific information. Scale selection is our answer to that warning.

The staging method is borrowed from Best of Both Worlds. That paper froze a pretrained speech encoder, trained the accent discriminator to proficiency, and only then started reversed-gradient training, reporting a 33% relative word-error reduction on unseen accents. It worked on one-dimensional audio with no spatial routing. CartoonGAN staged the generator rather than the discriminator, with a fixed 10-epoch cutoff. We replace fixed cutoffs with a criterion based on when the discriminator's accuracy plateaus.

The global module comes from CCNet. It attends along each pixel's row and column and repeats this twice to reach the whole image, using 11x less memory than non-local blocks.

Finally, the RGB-thermal fusion systems we must beat use no adversarial training at all. ABMDRNet reduces modality difference by translating features in both directions and adds multi-scale dilated context. LASNet uses depth-specific location, activation and edge modules. MixPrompt freezes an RGB backbone and injects auxiliary sensors as low-rank prompts.

Research Program

The architecture has three parts:

  • Streams: two ImageNet-pretrained ResNet streams, one for RGB and one for thermal.
  • Local fusion: low- and mid-level features are fused by simple channel-weighted addition feeding an edge-preserving decoder path. They receive no adversarial loss.
  • Global stage: each stream's deepest features pass through a shared recurrent criss-cross attention block with two recurrent steps. A convolutional modality discriminator $D$ reads these routed features through a gradient reversal layer.

The training objective is

$$\mathcal{L} = \mathcal{L}_{seg} + \lambda \, \mathcal{L}_{adv}\big(D(g(f_{rgb})),\, D(g(f_{th}))\big),$$

where $\mathcal{L}_{seg}$ is pixel cross-entropy on the fused prediction, $f_{rgb}$ and $f_{th}$ are the deepest stream features, $g$ is the criss-cross block, $\mathcal{L}_{adv}$ is the discriminator's per-location modality cross-entropy, and $\lambda$ is the reversal weight.

Phase one freezes both streams and $g$, taken from a model trained with $\mathcal{L}_{seg}$ alone, and trains discriminators only. We train several probe discriminators, each reading a different scale: a 1×1 discriminator on low-level features, one on dilated mid-level features, one on pre-attention deep features, and one on post-attention features. Their held-out accuracies, split by day and night, form a modality-identifiability profile across scales. This profile is a result in itself: it is the first direct measurement of where the sensing gap lives in an RGB-thermal network.

Phase two unfreezes everything and turns on the reversed gradient at the scale the profile supports. Phase one ends when the post-attention discriminator's validation accuracy changes by less than a small tolerance over a fixed window. This data-driven switch replaces a fixed epoch count.

We then run two robustness extensions that follow from the mechanism.

The first targets missing sensors. Following the missing-modality evaluation of the collapse paper, we drop RGB at test time on night images. If global alignment has made the thermal stream's context interchangeable with RGB's, performance should degrade gracefully.

The second targets imperfect registration. Side-by-side camera rigs mostly produce horizontal parallax, and criss-cross attention attends along rows. So we shift the thermal image horizontally by a range of pixel offsets and test whether the globally aligned model tolerates misregistration that MixPrompt, by its authors' account, does not.

Baselines

The naive reference is the same two-stream network with channel-weighted fusion and the criss-cross block, trained with segmentation loss alone. It shares every parameter and training step with the proposal except the discriminator. Any gain must show up against it, and it also isolates what global context contributes without alignment.

The strongest published competitors on MFNet are:

  • MixPrompt: reported at 61.0 mIoU under nighttime conditions in MixPrompt. The file does not say whether this is a night-only split, so we will rerun it rather than compare to the printed figure.
  • LASNet: reported at 54.9 mIoU and 75.4 mAcc in LASNet.
  • ABMDRNet: reported at 54.80 mIoU in ABMDRNet.

On PST900, LASNet reports 84.40 mIoU. The file has no reported MixPrompt or ABMDRNet number on PST900.

For fairness, we reproduce LASNet and MixPrompt from their descriptions on our splits. Where possible we match backbone family, input resolution, total training iterations and a fixed hyperparameter search budget, counting the phase-one discriminator iterations against our own budget. Beating them means exceeding our reproduced numbers on overall and night-split mIoU, averaged over five seeds, by more than the seed standard deviation. It also means not falling below the published figures on the benchmark as a whole.

Three ablations isolate the key ingredients:

  • Scale ablation: align everywhere (all probe scales reversed) and align only pre-attention local-to-deep convolutional features. This compares scale-selective against scale-agnostic alignment.
  • Variance-preserving version: align everywhere with the class-wise variance discrepancy term of Song et al. added. This tests whether their fix for over-stripping can stand in for scale selection.
  • Staging ablation: unstaged training from step zero, the standard smooth ramp of $\lambda$ from zero, and our frozen-feature warm-up. All three share the same total number of steps.

Why not just…?

Why not just skip adversarial alignment and use a strong fusion module or frozen-backbone prompting? This is the most serious competitor, since MixPrompt reports strong MFNet results with few trainable parameters and LASNet reaches 54.9 mIoU without any discriminator. The evidence for alignment is indirect. ABMDRNet's own ablation drops from 54.80 to about 51 mIoU without its (non-adversarial) modality-difference reduction, and the collapse paper argues that plain joint training starves the weaker modality. Our naive baseline and the rerun MixPrompt settle this directly. If alignment adds nothing on registered, complete inputs, we will report that, and the claim narrows to the missing-RGB and parallax conditions where prompting is known to be fragile.

Why not just ramp the reversal weight up from zero instead of pre-training the discriminator? A ramp is a soft version of staging and may work as well. It is a real threat to the staging component. The difference is that under a ramp the early gradients are small but still point in directions set by an untrained discriminator. After warm-up they are informative from the first step. That matters more after a global block, which spreads each gradient over the whole image. The three-way staging ablation decides this using seed variance, drift of the pretrained stream features during the first epochs, and final mIoU. If the ramp matches warm-up, we keep warm-up only as the measurement instrument and drop the stabilization claim. Scale-selective alignment remains the central claim either way.

Why not just align everywhere and protect sensor-specific information with a variance-preserving loss? Song et al.'s class-wise variance term does preserve identity spread across modalities. However, it constrains within-class variation, not the texture and temperature cues that separate classes at night. Keeping those cues is exactly what the scale choice is for. The align-everywhere-plus-variance ablation tests this. The comparison that decides it is per-class mIoU on night images for thermally distinctive classes.

Proposed Experiments

The main benchmarks are MFNet, with 1,569 registered day and night pairs and results reported overall and by day and night split, and PST900, recorded underground in darkness and smoke. Metrics are mIoU and mAcc over five seeds, plus per-class IoU. For robustness we report mIoU with RGB removed at test time and mIoU as a function of horizontal thermal offset. For training health we report seed variance and the relative change of pretrained stream features during the first training epochs.

Three checks decide whether the idea holds.

The first is the phase-one probe. If the post-attention discriminator is already near chance, there is nothing to align globally and the proposal fails at its premise. If it is near-perfect at every scale, the "layout is shared" intuition is wrong.

The second compares global-only alignment with align-everywhere and with the variance-preserving variant on night-split thermal classes. Global-only must win there, not merely tie overall.

The third compares warm-up, ramp and unstaged training after the global block. The prediction is that unstaged training hurts more with attention than without it, which we test by repeating the staging ablation with the criss-cross block replaced by a plain convolution of matched cost. If that interaction does not appear, the argument that global routing amplifies early discriminator noise is false, and we will say so.