Letting Geometry Decide Where: Gated Cross-Attention Proposals for Unseen Objects

Object detectors find objects in two steps. A proposal stage marks image regions that probably contain something, and a recognition stage names them. When the proposal stage is trained only on boxes for a fixed list of categories, it learns to treat everything else as background. Unseen objects are then dropped before any open-vocabulary classifier ever sees them. One partial fix is to train proposals on estimated depth and surface-normal maps instead of color images. Shape boundaries carry over to new categories, while color and texture shortcuts do not. That approach still runs geometry and color in separate offline stages, so the final detector cannot use color where geometry is unreliable. Geometry is unreliable on small and thin objects.

We propose a proposal network in which geometry asks the questions and color answers them. Features from monocular depth and surface normals act as the queries in a cross-attention layer, a mechanism where one stream selects which parts of another stream to read. Color-image features supply the keys and values. A learned per-location gate decides how much color evidence to mix back in. The central claim is about the direction of fusion. Because geometry queries color and the gate is regularized against color dominance, the fused network should keep geometry's recall on novel categories and recover the small-object recall that geometry alone loses. Naive fusion should instead fall back into the color "appearance trap": color is the more discriminative modality for the training classes, so the network learns to lean on it.

Why this, and not the obvious version

The obvious idea at this intersection is "fuse RGB and depth end-to-end inside the proposal network." It is not in the file. GOOD trains a proposal network on Omnidata depth and normals, uses its boxes as pseudo-labels, and then trains an RGB-only detector. Its authors note that geometry is never integrated end-to-end, and that depth is blurry for small or thin objects. OW-DETR and OrthogonalDet attack known-class bias with RGB-only features. OW-DETR mines unknowns from self-attention maps; OrthogonalDet separates objectness (feature magnitude) from class (feature direction). MAVL is the closest fusion work, and it fuses language with vision. MAVL uses late fusion and prompts such as "all objects," because early fusion smeared spatial detail. It therefore relies on text, which may pull boundaries toward familiar nouns. What Makes Good Open-Vocabulary Detector keeps the proposal stage purely visual and fuses with CLIP only in the second stage.

Plain RGB-depth fusion carries a risk that the file motivates but does not test. The literature on cross-modal fusion repeatedly reports modality dominance, where one stream washes out the other (Cross-Modal Attention Fusion for Heterogeneous Data Integration; the gated drug–target model Gated Cross Attention for DTI). In a proposal network trained on closed-set boxes, the dominant stream will be color, since color best separates the training classes from background. A symmetric fusion network could therefore score worse on novel recall than GOOD. Our contribution is the asymmetric, gated design, together with the experiments that show whether the asymmetry matters.

Core Mechanism

Let $G$ be multi-scale tokens from a small encoder over estimated depth and normals, and $R$ be tokens from the RGB backbone's feature pyramid. At each pyramid level the fused proposal feature is

$$F = G + g \odot \mathrm{DeformAttn}(Q{=}G,\; K{=}R,\; V{=}R), \qquad g = \sigma(W[G; \mathrm{DeformAttn}(\cdot)]),$$

where $\mathrm{DeformAttn}$ is multi-scale deformable attention. This is the sparse sampling attention MAVL used to make dense cross-modal attention tractable, so we reuse its technique. The gate $g \in [0,1]$ is computed per location, and $\sigma$ is the sigmoid. The gating form follows Gated Cross Attention for DTI, here with geometry as the preserved identity stream. The objectness and box heads read only $F$.

Two training choices keep color from taking over. First, supervision is ground-truth boxes plus GOOD-style geometric pseudo-labels, which removes the penalty on unlabeled objects (the background-suppression bias). Second, a gate prior adds a penalty $\lambda \, \mathbb{E}[g]$ wherever the depth map is locally confident. The gate thus opens mainly where geometry is unreliable, which we approximate by depth-gradient sharpness and region size. The gate prior is the key ingredient and is ablated directly. Depth and normals are predicted from the RGB image, so deployment needs no extra sensor, only the cost of the geometry estimator.

Research Program

We first reproduce GOOD's two-phase pipeline and the RGB-only proposal baseline in a shared codebase. We then train the fused network under the same schedule and data. We analyze it in three ways. Novel-object recall is broken down by object size, to test whether fusion recovers the small-object loss. The learned gate maps are inspected to test whether they open on small and thin objects and stay closed on large ones. The correlation between proposal objectness and known-class confidence is measured on novel objects, following OrthogonalDet's diagnostic for known-class bias, to test whether fusion has re-entered the appearance trap. Finally, the fused network replaces the proposal stage of an open-world detector, to test whether better proposals translate into higher unknown recall in the incremental setting.

Baselines

The do-nothing reference is a class-agnostic RGB proposal network trained only on base-class ground truth. This network shows the raw size of the novel-recall gap.

The strongest existing method on the core question is GOOD, which reports novel-object AR@100 of 25.9% in the COCO person-only setting. We rerun it on the same splits, backbone, geometry estimator and epoch budget. The fused model beats it if it improves novel AR@100 on both GOOD splits by a margin larger than the seed variance, and does not lose large-object recall.

Early fusion is a four-plus-channel RGB-depth-normal input, trained on identical pseudo-labels. It tests whether attention is needed at all.

Proposal union pools the boxes of GOOD's geometry network and an RGB network before non-maximum suppression, with the box budget matched. It is the most dangerous baseline, because it gets complementarity for free.

Two ablations isolate the design. A symmetric, ungated cross-attention variant and an RGB-as-query variant test the direction claim. Removing the gate prior isolates the anti-dominance term.

For the open-world front-end test, the reference is OrthogonalDet, which reports task-1 unknown recall on M-OWODB rising from 8.4% to 24.6%. RandBox reports 10.6% unknown recall, against 7.5% for OW-DETR. The fused proposals beat these references if they raise unknown recall when inserted into the same detector without lowering known-class mAP.

MAVL's class-agnostic VOC AP50 of 68.6 is reported only as context. It used 1.3M image-text pairs, so it is not a compute-matched comparison.

Why not just…?

Why not just stack depth and normals as extra input channels? This is cheap and may help. However, the tracking ablations in Frame–Event Tracking found concatenation clearly worse than cross-attention. Early stacking also gives the network no structural reason to prefer geometry for novel objects. Our early-fusion baseline, together with the objectness–class correlation diagnostic, will show whether stacking keeps GOOD's generalization or loses it to color.

Why not just keep GOOD's offline pseudo-labels and an RGB detector? GOOD already captures much of geometry's benefit. However, its final detector cannot consult geometry at test time, and its authors flag small and thin objects as a weakness. If our size-bucketed recall shows no gain on small novel objects over GOOD, the end-to-end claim fails and we will report that.

Why not just take the union of geometry and RGB proposals? This is a real threat. Union exploits complementarity without any training, at the cost of more boxes and duplicate scoring. We match the proposal budget (AR@100) so the comparison is fair. If union matches the fused network at equal budget, the honest conclusion is that learned fusion is unnecessary, and we would refocus on the gate maps as a reliability estimator.

Why not just pre-train a larger RGB proposal network? What Makes Good Open-Vocabulary Detector shows that diverse proposal pre-training helps. Labels for categories outside the taxonomy still become background, however, and GOOD shows that RGB models stay biased toward texture. We include a scaled RGB baseline trained with the same pseudo-labels, so that our gain is not simply a gain from extra pseudo-labels.

Why not prompt with language as MAVL does? Text adds a semantic stream that can bias boundaries toward nameable nouns, and it requires large image-text corpora. We do not run a matched comparison. Whether geometry and language queries are complementary is left for future work.

Proposed Experiments

The main benchmarks are COCO in the GOOD cross-category settings: train on person and evaluate on the other 79 classes, and train on VOC classes and evaluate on the non-VOC ones. We report novel AR@10, AR@100 and size-bucketed AR. LVIS rare categories test the long tail. UVO and ADE20K test dense recall; no reported result was found on Lacuna for GOOD's numbers on them, so we establish our own. M-OWODB and S-OWODB, with the fused network as the proposal stage of OrthogonalDet, report unknown recall, absolute open-set error and per-task mAP. We also report runtime including depth estimation, because RandBox's 10,000-box inference shows that efficiency claims matter.

Three checks are decisive. First, on small novel objects the fused network must beat GOOD, while on large novel objects it must not fall below GOOD. Second, the symmetric and RGB-query ablations must show higher correlation between objectness and known-class confidence, together with lower novel recall; this would confirm that direction and gating, not fusion per se, protect generalization. Third, at a matched box budget, the fused network must beat proposal union. Failing the third check would mean the mechanism adds nothing over pooling the two proposal sets.