Letting Relations Rescue Detections Lost to Domain Shift

Many vision systems describe a scene as a graph. A detector first finds objects, such as lanes, traffic lights, people or cups. A second stage then predicts how those objects relate, for example "this light controls that lane" or "this lane continues into that one." The weakness is that the second stage only sees what the detector hands it. If the detector misses an object, every relation involving that object is gone, and nothing downstream can bring it back. This "cascading error" problem gets much worse when the system is deployed somewhere it was not trained, such as a new city, new cameras, fog, or simulation-to-real. Under that kind of domain shift, detector confidence drops. Many real objects fall just below the confidence cutoff and are thrown away before relational reasoning ever runs.

We propose to keep those low-confidence candidates alive and let the relational stage vote them back in. The decision runs through global attention, a mechanism in which every candidate can look directly at every other candidate. A lane that is confidently detected and confidently predicted to continue implies that a successor lane exists. A confidently detected lane approaching a stop line implies a controlling sign. Our bet is that this kind of relational evidence transfers across domains better than raw appearance does, provided the relation representation itself is made domain-invariant. To that end, we apply adversarial domain adaptation not only to object features but to pairwise relation features. Adversarial domain adaptation trains a domain classifier to tell source from target, and trains the network to defeat that classifier. The deliverable is a measured answer to a narrow question: under shift, how many detector omissions can relational context recover, and does that recovery beat simply adapting the detector better?

Core Claim

Under domain shift, a large share of the entities the detector "misses" in structured scenes are not absent from its output. They are present as low-scoring candidates. A relation decoder that attends over all candidates, rather than only the top-scoring ones, and is adversarially aligned at the level of object pairs can raise the existence scores of candidates that have a confident relational partner. This should recover omissions that detector-level adaptation alone leaves on the table.

The claim makes a sharp mechanistic prediction. The recovered entities should be concentrated among those whose relational partner was detected, and the gain should grow with the size of the shift. If instead the gains are spread uniformly, or are just as large without the relation-level adaptation, the mechanism is not what we claim.

Where This Sits

Three lines of prior work motivate the proposal by documenting the gap. TopoMLP builds a detect-then-reason pipeline for driving road topology. Its ablations show that substituting ground-truth topology while keeping predicted detections barely moves the overall score, so detection is the bottleneck. CURVE makes driving scene graphs robust out of distribution through causal adjustment and pruning of uncertain edges. Its authors state that it cannot recover a vehicle the upstream detector missed. HostSG prunes noisy video scene graphs with an information bottleneck and acknowledges the same upstream ceiling. All three accept the detector's output as fixed. We do not.

Two works supply the methods we reuse.

  • Iterative feedback. Iterative Scene Graph Generation shows that separate subject, object and predicate decoders, refined over several steps with cross-attention between them, let relation predictions correct object predictions. It is evaluated only in-distribution, on Visual Genome and Action Genome. We take its feedback idea, apply it to existence scores of discarded candidates, and test it under shift.
  • Adversarial tokens. AQT adapts transformer detectors with a learnable adversarial token that attends over feature tokens to find domain-specific ones, aligning at patch, channel and object-query levels. It stops at boxes. We extend its adversarial token to pairs of object queries, so that the representation the rescue depends on is aligned.

Domain Adaptive Faster R-CNN adds image-level and instance-level discriminators with a consistency term to make region proposals domain-invariant. It constrains our design: alignment at a single global level is known to be insufficient, so we do not propose it.

One cited idea pulls against ours. CURVE prunes uncertain edges, while we promote uncertain nodes. These are compatible only if promotion is gated by a confident partner. We therefore treat hallucination, meaning promoting candidates that are not real objects, as a primary failure metric rather than an afterthought.

Research Program

We build on a TopoMLP-style pipeline because road topology gives dense, physically constrained relations. In this pipeline, a lane detector proposes 3D lane centerlines and a traffic-element detector proposes 2D signs and lights. We change three things.

Keep candidates. The detector keeps several times more candidates than usual, including low-score ones. Each candidate carries an existence logit.

Score feedback. A relation decoder attends over all candidates and predicts pairwise edge probabilities. Over $T$ refinement steps, each candidate's existence logit is updated from its relational support:

$$ z_i^{t+1} = z_i^{t} + g\Big(\sum_{j} a_{ij}^{t}\, p_{ij}^{t}\, \sigma(z_j^{t})\Big) $$

The terms are defined as follows:

  • $z_i^t$ is candidate $i$'s existence logit at step $t$.
  • $p_{ij}^t$ is the predicted probability of an edge between $i$ and $j$.
  • $a_{ij}^t$ is the attention weight $i$ places on $j$.
  • $\sigma$ is the sigmoid, so $\sigma(z_j^t)$ is $j$'s current existence confidence.
  • $g$ is a small learned network.

Support therefore flows only from candidates that are themselves believed to exist.

Pair-level adversarial alignment. On top of AQT-style object-query alignment, an adversarial token attends over pairwise relation features $[q_i, q_j, q_i \odot q_j]$, where $q_i$ and $q_j$ are the two object queries. Its job is to predict the domain, and a gradient reversal layer trains the relation features to fool it. This targets the relational topology directly. It follows the observation that domain shift in road scenes includes regional layout and camera geometry, not just texture, as noted in the OpenLane-V2 benchmark description.

Supervision. On source data, training uses standard bipartite matching. Matched candidates are pushed toward existence and unmatched ones away, so the feedback learns not to promote clutter. Target data is unlabeled and enters only through the discriminators.

Analysis. We add a diagnostic split that the field currently lacks. Each target-domain ground-truth entity that the source-only detector misses at its standard threshold is labeled by two properties:

  • whether it appears among the kept low-score candidates, which measures the recoverable pool;
  • whether at least one of its ground-truth relational partners was detected.

We report recovery rate within each cell. We also report hallucination rate, defined as the fraction of promoted candidates that match no ground truth.

Baselines

Do nothing. The naive reference is source-only TopoMLP trained on the source domain and evaluated on the target with its standard top-K selection. It shows how much shift costs. We also run it in-domain, to anchor against TopoMLP's reported 41.2% OpenLane-V2 Score (OLS) on Subset_A with a ResNet-50 backbone (TopoMLP). No reported result was found on Lacuna for cross-subset transfer on OpenLane-V2.

Adapted detector, unchanged reasoning head. This is the strongest competitor: TopoMLP whose detectors are adapted with full AQT, with the topology head unchanged. Since TopoMLP's own ablation says detection is the bottleneck, simply adapting the detector is the most plausible way to capture most of the available gain. AQT's reported detection-adaptation strength is 47.1 mAP on Cityscapes→Foggy Cityscapes and 53.4 car AP on SIM10K→Cityscapes (AQT). No reported result was found on Lacuna for AQT on OpenLane-V2 or on any relational task. It will use the same backbone, the same unlabeled target images, the same number of training iterations, and the same hyperparameter search budget as our method. Beating it means a statistically reliable OLS gain on the target over three seeds, together with a higher recovery rate in the "partner detected" cell and no worse hallucination rate.

Non-adversarial invariance. CURVE-style uncertainty pruning plus invariant regularization, which reports 35.07 MCC versus 27.94 for plain IRM on zero-shot DeepAccident (CURVE). We use it on our secondary benchmark, where those numbers apply.

Three ablations isolate the key ingredients.

  • Many candidates, no feedback. This tests whether merely passing more candidates to the head explains the gain.
  • Feedback without pair-level adaptation. This tests whether relation features need alignment.
  • Feedback trained on source only. This tests whether the relational prior transfers without any adaptation.

Why not just…?

Why not just adapt the detector with AQT and keep the reasoning head as is? This is the real threat. TopoMLP's evidence that detection dominates suggests it could capture most of the gain. Our claim survives only if relational rescue recovers entities that the adapted detector still misses. The diagnostic split settles it. If AQT-plus-TopoMLP already empties the "missed but partner detected" cell, our mechanism has nothing left to do, and we would report that as the finding.

Why not just lower the detector's threshold or keep more candidates? This is cheap and may help recall. But Zhou et al. document that confidence thresholds trade false negatives against clutter. TopoMLP notes that false positives with inflated confidence can game scoring. The "many candidates, no feedback" ablation directly measures how much of our gain is just a looser threshold. We expect that ablation to raise hallucination alongside recall, whereas partner-gated feedback should not.

Why not just self-train on target pseudo-labels? Self-training means using the model's own confident target predictions as labels. It is the standard non-adversarial approach in domain adaptation, but no reported result was found on Lacuna for it in this setting. Its weakness here is structural: pseudo-labels come from the same threshold that drops the omitted entities, so it tends to reinforce omissions. We run it as an additional baseline under the same budget. We also test whether combining it with our feedback is additive, which would suggest the two address different failures.

Why not just use causal or invariant-risk methods instead of adversarial training? CURVE outperforms IRM. But it operates downstream of fixed detections, and its authors state it cannot restore missed entities. On CARLA-SR→DeepAccident we compare against CURVE's reported 35.07 MCC by feeding CURVE's relational head with our rescued candidates versus its original detector proposals. This tests whether rescue is complementary to invariance-based pruning.

Proposed Experiments

Primary benchmark. The primary setting is OpenLane-V2, training on Subset_A (Argoverse 2, seven cameras) and adapting without labels to Subset_B (nuScenes, six cameras). This is a natural shift in city, sensor rig and layout. Metrics are OLS and its components: lane detection, traffic-element detection, lane-to-lane topology and lane-to-element topology. The differing camera counts are a real engineering risk; we handle them by operating in a shared bird's-eye-view query space. We also run B→A to check symmetry.

Secondary benchmark. The secondary setting is CARLA-SR→DeepAccident (zero-shot) and CARLA-SR→DoTA, using CURVE's protocol and MCC.

Upper bound. Following TopoMLP's logic, we run an oracle analysis on the target domain with ground-truth detections plus predicted topology. This bounds what any rescue could achieve.

Decisive checks. The proposal succeeds only if all of the following hold:

  • Target-domain gains over AQT-plus-TopoMLP are concentrated in entities whose relational partner was detected, with little change for isolated entities.
  • The gain from feedback is small in-domain (A→A) and larger under shift (A→B).
  • Removing pair-level adaptation shrinks the gain under shift but not in-domain.
  • Hallucination rate on the target stays at or below that of the "many candidates" ablation.

If the first check fails, relational context is not doing the rescuing. If the third fails, the adversarial component is unnecessary, and the proposal reduces to a simpler, still useful in-distribution feedback method.