Networks that combine a color image with a second sensor, such as depth or thermal, usually fuse the two streams in one of two ways. At low resolution they use attention, where features from one sensor pick out relevant features from the other. At high resolution they fall back to plain concatenation, because attention over every pair of pixels is too expensive there. This is backwards from where the sensors actually disagree. Depth is usually trustworthy about coarse layout ("there is an object here") and untrustworthy at fine scale, where its edges are blurred, shifted, or missing. Color has crisp edges but is easily fooled by texture and lighting. The decision about which sensor to believe matters most at object boundaries and at the finest resolutions, which is exactly where current models stop making it.
We propose to run cross-modal attention only where it is needed: a thin band of uncertain pixels around the boundaries of the coarse prediction, at each finer scale. At every pixel in that band, a query built from the coarse fused representation looks into a small local window of each sensor's high-resolution features. A learned per-pixel gate then decides how much of each sensor's evidence to admit, or whether to keep the current feature unchanged. Two predictions follow. Because the band's size grows with boundary length rather than image area, cost stays manageable at full resolution. And because the coarse prediction already encodes object-level agreement between sensors, the fine-scale arbitration can be local, which also absorbs small misalignments between the sensors.
Core Claim
Coarse-to-fine refinement and cross-modal attention should not be separate stages. The coarse fused prediction should decide where high-resolution cross-modal attention runs and what it queries for. Our specific claim is falsifiable. Under matched compute, a decoder that applies gated, windowed cross-modal attention only on the uncertain boundary band at each scale will beat (a) per-scale concatenation, (b) dense within-scale cross-modal attention at every pixel, and (c) unimodal color-only boundary refinement. The largest gains should appear on boundary-sensitive metrics and under depth that is degraded or misaligned.
Where This Sits in Prior Work
Several RGB-D and RGB-thermal models already mix multi-scale decoding with cross-modal attention. They all compute fusion uniformly over the grid and at matched scales.
- HiDAnet runs bidirectional channel and spatial cross-attention between color and depth at each matched encoder level. It slices depth into discrete planes with hand-set thresholds, and a coarse-to-fine decoder then merges the results.
- CAE-Net uses cross-attention units for RGB-thermal fusion. To keep each sensor's details from being washed out, it maintains three parallel decoders, and it reports a collapse when its VGG-16 backbone is swapped for ResNet-50.
- RDFNet fuses color and depth at four backbone stages with static residual convolution blocks, so it has no input-dependent trust at all.
None of the three lets the coarse prediction steer where or how fine-scale fusion happens. We reuse their matched-level encoders. What we replace is the fine-scale fusion rule.
Two papers motivate the efficiency side directly.
- Learning Selective Mutual Attention and Contrast for RGB-D Saliency Detection builds mutual attention in which one sensor's attention map guides the other. Its authors state that computing this at high resolution is prohibitive, so they restrict it to deep layers and use naive fusion in shallow stages. That gap is precisely what we target.
- Pang et al. show in unimodal CT segmentation that active attention on every skip connection improves boundaries at small parameter cost. They also flag the dense-grid cost of that attention.
Two prior efficiency strategies exist, and we differ from both.
- RIOcc uses sparse deformable attention between camera and LiDAR in a bird's-eye-view grid. It is still applied everywhere on the grid and only down two scales.
- The encoder-fused referring network of Referring Segmentation via Encoder-Fused Cross-Modal Attention Network avoids the quadratic cost because one modality is a short word sequence. That trick does not apply when both inputs are dense images.
The idea closest in spirit to coarse-guided refinement is RefineFuse. It passes deep cross-attended semantics down into shallow streams, but it uses a different, non-attention mechanism at shallow layers. It applies that mechanism to every pixel, and it outputs a fused image rather than a mask.
Our method borrows from three further sources.
- Masking fine features by deep evidence. SISE masks early high-resolution features with deep semantic activations in a single-sensor setting.
- Revisiting a local window. Interactive Segmentation with Elaborate Focus Prior revisits a local window around user clicks after a global prediction. We replace the click with model uncertainty.
- Gated residual update. An Interpretable Framework for Drug-Target Interaction with Gated Cross Attention shows that ungated cross-attention lets a dominant input pull the other's features far from their original values, and that a learned gate between the attended context and the original feature prevents this. We adopt the gate because a cascade of four or five refinement stages compounds exactly this drift. That compounding is why CAE-Net needed separate per-sensor decoders.
The Operator
Let the decoder hold a feature map $h^{(s)}$ and a coarse prediction $P^{(s)}$ at stride $s$. Upsampling to stride $s/2$, we select the uncertain set $U^{(s/2)}$: the pixels whose upsampled prediction entropy is in the top fraction $\rho$. In practice this is a band a few pixels wide around predicted boundaries. All other pixels receive ordinary upsampling plus a 1×1 convolution on concatenated features, so they are treated exactly like the concatenation baseline.
For each $u \in U$, the query $q_u$ is a projection of the upsampled decoder feature at $u$. Keys and values come from the color features $F^{R}$ and the depth (or thermal) features $F^{D}$ in a $k \times k$ window $W_u$ centered at $u$ at the current resolution. The update is
$$h_u \leftarrow h_u + g^{R}_u\,\mathrm{Attn}(q_u, F^{R}_{W_u}) + g^{D}_u\,\mathrm{Attn}(q_u, F^{D}_{W_u}),$$
where $\mathrm{Attn}$ is standard scaled dot-product attention restricted to the window, and $g^{R}_u, g^{D}_u \in [0,1]$ are sigmoid gates predicted from $q_u$ and the two window summaries. Both gates can go to zero, which leaves $h_u$ unchanged. This is the residual identity path that guards against drift.
The window serves two purposes. It lets a depth edge that is offset by a few pixels still be matched, and it bounds the cost per selected pixel at $O(k^2 d)$, where $d$ is the feature width. Total cost per scale is $O(|U|\,k^2 d)$. For compact objects, $|U|$ grows with perimeter times band width, roughly $\sqrt{HW}$, rather than with area $HW$. This is the source of the efficiency claim, and we will measure it rather than assume it.
The coarse query carries object-level agreement between the sensors down to the fine scale. We consider this the essential difference from matched-level fusion: the high-resolution decision is conditioned on what the fused coarse representation believes the object is.
Research Program
We will build the decoder on a standard two-stream encoder, implemented on both VGG-16 and ResNet-50, because CAE-Net reports a backbone collapse and we want to know whether a local, gated operator avoids it. Refinement runs from stride 32 down to stride 4 or full resolution, with deep supervision at each stage.
The analysis has three parts.
- Cost. We measure actual memory and FLOPs against image resolution to test the perimeter-scaling prediction.
- Gate behavior. We record the gates by scale and by depth quality, to test the premise that depth is trusted at coarse scales and distrusted at boundaries when it is noisy. If the gates do not show this pattern, the motivating story is wrong, even if accuracy improves.
- Feature drift. We track cosine similarity between each sensor's features before and after refinement across stages, following the drift diagnostic of the gated cross-attention paper. This lets us confirm that a single decoder does not collapse toward one sensor.
Baselines
Naive reference: per-scale concatenation. A two-stream encoder whose features are concatenated at each scale and decoded by a standard top-down feature pyramid. It uses the same backbone, schedule, augmentation, and hyperparameter search budget as our model. It anchors every comparison, because on unselected pixels our model reduces to it.
Strongest RGB-D segmentation reference. For segmentation, the best reported number in the file is RDFNet at 50.1% mIoU on NYUDv2 (against 46.5% for color alone) and 47.7% mIoU on SUN RGB-D. We will cite these numbers but not compare to them directly, since backbones and training differ. Instead we reimplement its residual multi-level fusion on our backbones.
Strongest RGB-D saliency reference. For saliency, HiDAnet is the closest architecture: matched-level cross-attention plus a coarse-to-fine decoder. It reports 0.062 MAE on COME15K Difficult. Bilateral Attention Network for RGB-D Salient Object Detection reports 0.915 S-measure on NJU2K, and SMAC reports 0.790 maxF on ReDWeb-S. We will run HiDAnet's design with our training recipe and backbone so that the only difference is the fine-scale fusion rule.
RGB-thermal reference. On VT5000, CAE-Net reports only relative gains over CSRNet: 10.0% lower MAE and a 1.7% higher F-measure. We found no reported absolute number on Lacuna for VT5000, so we will retrain CAE-Net ourselves.
Strongest competitor and its ablation. The strongest competitor on our specific question is dense gated cross-modal attention at every pixel of every scale, using the same windowed operator but without uncertainty selection. Where dense application exceeds memory at fine scales, we instead raise the window or band budget of our model to match the dense model's FLOPs. This ablation isolates whether selection matters, or whether windowed gated attention everywhere does as well.
Further ablations. Three more variants each remove one ingredient:
- a coarse query versus a same-scale query, which tests whether coarse guidance matters;
- gate removed;
- window size $k=1$, which removes misalignment tolerance.
What counts as winning. Beating a baseline means a higher mean over three seeds, with non-overlapping seed ranges, on boundary F-score and on the corruption suites described below, at equal or lower FLOPs.
Why not just…?
Why not just concatenate at every scale of a feature pyramid? Prior ablations argue against it under noise. CAE-Net reports that concatenation and addition are misled by degraded thermal or dark color input. RDFNet found concatenation fails on ambiguities such as white objects on white backgrounds. Those results come from clean or mildly noisy data, though. On clean NYUDv2, concatenation may close much of the gap. The corruption suites are where we expect separation, and if none appears there, the method has no case.
Why not just run efficient cross-modal attention everywhere? This is the real threat. Deformable or windowed attention at every pixel, as in RIOcc, might match us, with selection adding nothing but complexity. The FLOP-matched dense ablation settles it. If dense windowed attention equals our model at equal compute, we will report that the useful ingredient is the gated, coarse-queried window, and drop the selection claim. Our prediction is that the dense model spends its budget on interior pixels where the sensors already agree, and so loses at matched cost on boundary metrics.
Why not just fuse once at the bottleneck and refine boundaries with color alone? Color edges are sharp, so perhaps depth contributes nothing below stride 16. RefineFuse reports blurred edges when shallow interaction is dropped, and the encoder-fusion ablation in Referring Segmentation via Encoder-Fused Cross-Modal Attention Network shows a 3.3% J&F gain from fusing inside the hierarchy. Neither isolates fine-scale depth for dense RGB-D masks, however. We test this directly with a variant that applies the same selected-band refinement using color keys only. If that variant matches ours, the honest conclusion is that cross-modal fusion belongs only at coarse scales.
Why not just reweight depth globally by an estimated quality score? Per-image depth quality weighting, as in SMAC's selective attention, cannot express the case where depth is good for layout and bad at edges within the same image. We compare against a global quality-gated variant. The per-scale gate statistics will show whether the trust signal really varies spatially and by scale.
Proposed Experiments
Datasets. Segmentation uses NYUDv2 and SUN RGB-D, reporting mIoU, boundary F-score within a narrow trimap, and Hausdorff distance. Saliency uses NJU2K, COME15K, and ReDWeb-S, reporting MAE, S-measure, maxF, and boundary F. ReDWeb-S is especially relevant because its depth is estimated rather than measured. RGB-thermal uses VT5000, VT821, and VT1000.
Decisive checks. Three experiments decide the outcome.
- Controlled corruption suite. On each test set we shift depth by 2 to 8 pixels, blur its edges, add holes, and add thermal flares. The prediction is that our model's advantage over dense attention and concatenation grows with shift magnitude up to roughly the window radius, then flattens.
- Resolution sweep. Memory and FLOPs from 480p to 1080p should grow sub-linearly in pixel count for our model and linearly for dense attention.
- Backbone swap. Moving from VGG-16 to ResNet-50 should not produce the drop CAE-Net reports.
If the first check shows no corruption-dependent gain, and the color-only selected-band variant matches ours, we will conclude that fine-scale cross-modal attention is unnecessary for these tasks, which would itself be a useful negative result.
References
- HiDAnet: RGB-D Salient Object Detection via Hierarchical Depth Awareness
- CAE-Net: Cross-Modal Attention Enhancement Network for RGB-T Salient Object Detection
- RDFNet: RGB-D Multi-level Residual Feature Fusion for Indoor Semantic Segmentation
- Learning Selective Mutual Attention and Contrast for RGB-D Saliency Detection
- Correlation Matters: Multi-scale Fine-Grained Contextual Information Extraction for Hepatic Tumor Segmentation
- RIOcc: Efficient Cross-Modal Fusion Transformer with Collaborative Feature Refinement for 3D Semantic Occupancy Prediction
- Referring Segmentation via Encoder-Fused Cross-Modal Attention Network
- RefineFuse: an end-to-end network for multi-scale refinement fusion of multi-modality images
- Explaining Convolutional Neural Networks through Attribution-Based Input Sampling and Block-Wise Feature Aggregation
- Interactive Segmentation with Elaborate Focus Prior
- An Interpretable Framework for Drug-Target Interaction with Gated Cross Attention
- Bilateral Attention Network for RGB-D Salient Object Detection