Asking Depth Only Where the Eye Is Unsure: Uncertainty-Routed Cross-Attention for Camouflaged Object Detection

Camouflaged object detection means segmenting animals or objects that match their background in color, texture and pattern. It is hard because the local edges that ordinary segmentation networks rely on almost disappear at the object's outline. A natural fix is to add a second signal that camouflage does not fool, such as scene depth: a moth on bark matches the bark's color, but a lizard on a branch still sits in front of the leaves behind it. Prior work found that this second signal cuts both ways. Depth predicted from a single RGB image by an off-the-shelf estimator is noisy on camouflage images, and fusing it naively makes results worse (Depth-Guided Camouflaged Object Detection).

We propose that the RGB network should consult depth only at the pixels where it is itself uncertain, and only to the extent that the depth looks trustworthy there. Concretely, a coarse RGB prediction marks a thin band of ambiguous pixels, usually around the true boundary. Only the image features in that band attend to nearby depth features through cross-attention, meaning one modality's features act as queries that select and weight features of the other. A learned gate then decides how much of the retrieved depth evidence to mix in. Confident interiors and clear background never see depth, so depth noise cannot corrupt them. Because only a small fraction of positions query depth, the cost stays close to that of an RGB-only model, and everything runs in one forward pass.

Core Claim

The claim has two parts, and each can be tested separately. The first is accuracy. Routing cross-modal attention by the RGB model's own single-pass uncertainty should beat dense cross-attention and loss-level depth weighting. The gain should show up mainly on boundary-ambiguous images, and the model should hold up much better as depth quality degrades. The second is cost. This accuracy should come at a small fraction of the compute of dense cross-attention, which scales with the product of the two token counts. A reviewer would also ask whether depth helps at all or merely brings in extra pretrained capacity. We treat that as a first-class question with a dedicated control, described below. If depth adds nothing beyond capacity, the proposal fails and we will report that.

The idea sits between two existing lines of work. CamoFormer splits self-attention heads into foreground, background and global groups using coarse masks from earlier decoder stages. That shows mask-restricted attention reduces background contamination, but it works only inside RGB. Depth-Guided Camouflaged Object Detection motivates our design: it found that pseudo-depth from MiDaS is a "double-edged sword" and down-weighted bad depth regions at the level of the loss, using GAN-produced confidence maps. Depth never directly shapes individual RGB features there. The gating mechanism comes from Gated Cross Attention for drug–target interaction, which mixes cross-attended context with the original features through a learned gate so that the original representation is preserved. We reuse that residual gate but decide where it applies by uncertainty.

Frequency- and text-based fusion (SFGNet, FGSA-Net) apply cross-domain attention over the whole image and to signals derived from the same RGB input. We instead target an external geometric signal and restrict fusion spatially. Uncertainty-aware models such as UGTR estimate uncertainty with 50 sampled forward passes. We use a deterministic proxy and spend the uncertainty on deciding where fusion happens rather than on masking during training.

Research Program

The model has an RGB encoder–decoder with a standard backbone, a depth encoder over MiDaS pseudo-depth (following Depth-Guided COD), and a routed fusion block at each decoder scale. At each scale the decoder emits a coarse foreground probability $p_i$ per position $i$, and we define a cheap uncertainty $u_i = 1 - |2p_i - 1|$, which is 1 where the prediction is a coin flip and 0 where it is certain. The routed set $R$ is the fraction $\rho$ of positions with the highest $u_i$. We choose top-$\rho$ rather than a threshold so that compute is fixed. Each routed RGB feature $h_i$ attends only to depth features $d_j$ in a local window $W(i)$, and is updated as

$$h_i' = h_i + g_i \odot \mathrm{softmax}\!\left(\frac{(W_q h_i)(W_k d_{W(i)})^\top}{\sqrt{d_k}}\right) W_v d_{W(i)}, \qquad g_i = \sigma\big(\mathrm{MLP}[h_i, c_i]\big),$$

where $W_q, W_k, W_v$ are learned projections, $d_k$ is the key dimension, $c_i$ is the attended depth context (the softmax term), and $g_i$ is a per-channel gate in $[0,1]$. Positions outside $R$ pass through unchanged. The cost per scale is proportional to $\rho \cdot HW \cdot |W|$, where $HW$ is the number of image positions and $|W|$ the window size. Dense cross-attention costs $(HW)^2$.

We make three choices explicit and ablate each. The first is whether routing is hard top-$\rho$ or soft, with $u_i$ multiplying the gate. The second is whether the gate also sees a depth-reliability feature, namely local agreement between depth edges and RGB edges. The third is whether routing is learned end to end or fixed to the uncertainty proxy. To guard against the gate saturating to zero, a known failure of gated fusion that would silently turn the model back into RGB-only, we log gate distributions throughout training.

The analysis asks when depth should help. Many camouflaged animals, such as flatfish on sand or moths on bark, lie flush against their background and create no depth discontinuity. We therefore predict gains concentrated on COD10K's "Indefinable Boundaries" and "Occlusion" attribute subsets, and little or no gain on flat-lying categories. We also predict that the learned gate will be low on those flat-lying images. If the gate is not selective in this way, that counts against the mechanistic story even if average scores improve.

Baselines

The naive reference point is the same RGB encoder–decoder with the depth branch removed. It is trained on the standard COD10K (3,040) plus CAMO (1,000) training images and tested on COD10K (2,026), CAMO (250), NC4K (4,121) and CHAMELEON (76). Every other variant must beat it on S-measure, weighted F-measure, mean E-measure and MAE, and not merely match it.

The strongest competitor on the specific question of using pseudo-depth is Depth-Guided Camouflaged Object Detection. It reports an S-measure of 0.829 on COD10K with a Res2Net50 backbone. We will match its backbone, depth source (MiDaS) and training data, and rerun it under our own schedule. Beating it means a higher COD10K S-measure than both 0.829 and our rerun, with the gap coming mostly from the boundary-attribute subsets. The strongest RGB-only attention method with a reported number in our file is CamoFormer, with a weighted F-measure of 0.786 on COD10K. The file does not state its backbone, so we will retrain it on our backbone for a matched comparison and report the published figure separately.

SFGNet reports S-measure 0.912 and F-measure 0.884 on NC4K. It uses CLIP backbones and class-name text prompts, so we list it as a reference with more input information rather than as a like-for-like competitor. For FGSA-Net the file gives only relative gains over UEDG, so no absolute baseline number is available here.

Three fusion ablations isolate the key ingredient under an identical backbone, depth encoder, parameter count within 5%, and hyperparameter search budget. The first is early fusion, with depth as a fourth input channel. The second is dense gated bidirectional cross-attention over all positions, modeled on the Cross-Attention Blocks of CDFI. The third is routed attention with random routing at the same $\rho$. The random-routing variant is the ablation that directly tests whether uncertainty-based selection matters.

Why not just…?

Why not just feed depth as a fourth input channel? It is the simplest option, and Depth-Guided COD reports that standard fusion of pseudo-depth fails on camouflage because estimator noise leaks into every feature. We do not assume that result carries over. Early fusion is a trained baseline in our study, and the depth-corruption sweep below will show whether it degrades faster than routed fusion.

Why not just use dense gated cross-attention everywhere? This is the real threat. A well-tuned gate might learn on its own to ignore depth where it is not needed, which would make routing only a speedup. In frame–event tracking, the dense bidirectional cross-attention of CDFI beat concatenation by 10.4% in Success Rate on FE108, so dense cross-attention is a strong design. If dense and routed variants tie on clean depth and on the corruption sweep, we will present the contribution as matching accuracy at much lower compute, which ERRNet's 79.3 FPS shows matters for deployment. We will not claim an accuracy mechanism in that case.

Why not skip depth and use a stronger RGB model? Pseudo-depth is computed from the same RGB image, so any gain might just reflect MiDaS's pretrained capacity rather than geometric information. To test this, we replace the depth input with features from the MiDaS encoder taken before its depth head, and separately with depth maps from mismatched images, keeping the architecture fixed. If the pre-head features perform as well as depth maps, the geometric story is wrong. FGSA-Net's finding that frozen SAM or ViT backbones struggle with low-contrast camouflage boundaries suggests that capacity alone is not sufficient, but only this control can decide the question.

Proposed Experiments

The main table reports the four standard metrics on all four test sets, plus frames per second and FLOPs on one GPU for every baseline and ablation. The COD10K attribute breakdown tests the prediction that gains concentrate on "Indefinable Boundaries" and "Occlusion" and are near zero on flat-lying categories.

The first decisive check is a depth-corruption sweep. We add increasing Gaussian noise, blur and random region dropout to the pseudo-depth at test time and plot S-measure against corruption strength for early fusion, dense gated attention and routed attention. The routed model should lose the least, because unrouted positions are immune to depth. The second decisive check is a $\rho$ sweep from 5% to 100%, where 100% recovers dense attention. It shows whether accuracy saturates at a small routed fraction. The third check is the capacity control from the previous section. Together these separate the three possible outcomes: routing helps accuracy, routing only saves compute, or depth itself is a placebo.