Spending Extra Positive Matches Where the Domain Shift Is

Detection transformers (DETR, detectors that predict a fixed-size set of boxes and learn by pairing each true object with exactly one prediction) learn slowly because almost every prediction is told "background" on every step. Domain adaptation makes this worse. When a detector trained on clear-weather or synthetic images is adapted to foggy or real images without labels, the only supervision on the new domain comes from a teacher model's pseudo-labels, which are sparse and noisy. Existing adaptive transformers still pair each pseudo-label with exactly one student prediction. We propose to give each target-domain object a variable number of extra positive matches during training. The count grows with how strongly that object's image region has shifted between domains, is capped by how trustworthy its pseudo-label is, and shrinks back to one as the student catches up. Inference is unchanged: one set of queries, no duplicate-removal step.

The motivation comes from two findings that have not yet met. Work on one-to-many assignment shows that extra positive matches speed up and improve transformer training, but gives every object the same number of extras. Work on local domain imbalance shows that shift is concentrated on particular objects, such as small pedestrians and riders in fog, and that spreading adaptation effort evenly wastes it on easy regions. The claim to test is that, for a fixed total budget of extra positive matches, allocating them by local shift beats allocating them uniformly, and that the gain appears specifically on high-shift objects.

Where this sits

Three transformer adaptation methods keep strict one-to-one matching. DATR aligns class prototypes pooled from decoder queries across domains. DA-DETR fuses convolutional and transformer features and aligns them with one global discriminator; its authors name query-level adaptation as future work. PM-DETR retrieves domain prompts using image-level similarity. None of them changes how pseudo-labels are matched to queries, and none weights regions by local shift. These papers define the setting and the baselines. We keep their teacher-student structure and change only the matching.

The one-to-many methods supply our mechanism, all in single-domain training. Group DETR runs $K$ query groups that share weights. Each group does its own one-to-one match, self-attention is masked within groups, and only the first group is kept at test time. Co-DETR adds dense auxiliary heads and reports that too many of them create conflicting gradients. Semi-DETR applies one-to-many matching to noisy pseudo-labels early in training and switches to one-to-one at a hand-set iteration. It filters pseudo-labels with a Gaussian mixture fitted to matching costs. Sparse Semi-DETR uses top-$k$ filtering instead. We reuse Group DETR's masked groups and Semi-DETR's cost-based reliability estimate. We differ from both in two ways: multiplicity is set per object rather than globally, and the return to one-to-one matching is driven by the data rather than a fixed schedule.

The local-imbalance literature supplies the signal, and so far it has been tied to two-stage convolutional detectors. Differential Alignment upweights instances where teacher and student disagree and splits foreground from background in adversarial alignment on Faster R-CNN. Its ablation shows that background should be down-weighted but not discarded. DSD-DA uses a pixel-level domain discriminator to find target-relevant regions. It also shows that shift worsens the mismatch between classification confidence and localization quality. Harmonizing Transferability and Discriminability shows that aligning uninformative background causes negative transfer. These papers apply local shift to alignment loss weights. We apply it to match multiplicity, which is a different lever in a different architecture.

One finding constrains the design. DATR reports that filtering queries by confidence or by Hungarian matching hurt its prototype alignment. We therefore never filter or alter the primary query group. All shift-dependent behaviour lives in auxiliary groups that are discarded at inference.

Core mechanism

The base is a Deformable-DETR teacher-student adapter built in the ALDI codebase (ALDI), which supports DETR. We add $K_{\max}$ auxiliary query groups in the Group DETR style. For each target pseudo-box $i$, we compute two scores.

The first is a local shift score $s_i \in [0,1]$. We obtain it by pooling a lightweight pixel-level domain discriminator's output over the box, in the spirit of DSD-DA's discriminator but used only during training. We then average this with the normalized squared teacher-student class disagreement, following Differential Alignment.

The second is a reliability score $r_i \in [0,1]$: the posterior of the "clean" component of a two-component Gaussian mixture over teacher matching costs, as in Semi-DETR.

The number of groups in which object $i$ appears as a target is

$$ k_i = 1 + \left\lfloor (K_{\max}-1)\, s_i\, r_i \right\rfloor . $$

Here $k_i$ counts the primary group plus the first $k_i-1$ auxiliary groups. Each group still runs an ordinary Hungarian match, but only against the objects assigned to it. The multiplication by $r_i$ matters. High-shift objects are exactly where the teacher is least reliable, and multiplying supervision on a wrong box would amplify its error.

Because $s_i$ includes teacher-student disagreement, $k_i$ falls toward one as the student learns an object. This replaces Semi-DETR's fixed switch point with a per-object transition. Source images, which have clean labels, use a fixed small multiplicity. The design question there is only whether source-domain extras help or dilute the target signal, which we test.

Baselines

The naive reference is source-only Deformable-DETR evaluated on the target, plus the same teacher-student adapter with pure one-to-one matching. This is roughly DATR's supervision regime without its prototype modules. Every other variant is measured as a gain over this adapter under an identical schedule and seeds.

The strongest transformer competitor is DATR. On Cityscapes→Foggy Cityscapes it reports over 52.8% mAP, on Sim10k→Cityscapes over 66.3%, and on Cityscapes→BDD100k 41.9%. We rerun it in our codebase with its released settings where possible, and we also add our assignment on top of it, since the two are compatible. The strongest overall number in the file is Differential Alignment's 57.3% mAP on Foggy Cityscapes (with 45.8% on BDD100k and 69.7% on Sim10k), obtained with Faster R-CNN. We report it as the reference to beat but do not claim a controlled comparison, because the backbone and detector differ.

ALDI++ with its DETR configuration is the modern self-training baseline. The file reports only relative gains for it (+3.5 AP50 on Foggy Cityscapes, +5.7 AP50 on Sim10k), so we will run it ourselves. Following ALDI's warning about inconsistent protocols, every method gets the same backbone, image resolution, iteration count, and a fixed tuning budget of the same number of trial configurations on a held-out split of target images with labels hidden during training. We report AP50, the metric the adaptation papers call mAP, alongside COCO-style AP.

The central ablations isolate the key ingredient. The first is uniform one-to-many: every object gets the same $k$, chosen so that the total number of positive matches per epoch equals our method's. The second is one-to-one matching with each object's loss scaled by $k_i$, which gives the same gradient weight with no extra queries. The third is shift-free allocation, where $s_i$ is replaced by a random score with the same distribution or by its inverse. The fourth is reliability-free allocation, with $r_i=1$.

Beating the baselines means two things. The method must exceed uniform one-to-many at matched budget by more than seed variance, averaged over three seeds on at least two of the three benchmarks. It must also reach or exceed DATR's reported numbers when both run in our codebase.

Why not just…?

Why not just use uniform one-to-many matching, as in Group DETR or Semi-DETR, inside a domain-adaptive teacher-student? This is the real threat, and it is quite possibly where most of the gain over one-to-one training comes from. The file contains no reported result on any adaptation benchmark for uniform one-to-many DETR, so the matched-budget ablation above is the settling experiment. If uniform allocation ties ours overall and on high-shift objects, we will report the simpler finding: dense matching is the missing ingredient in adaptive transformers and local shift adds nothing to it. That is still a useful result, and it would redirect the local-imbalance line away from assignment.

Why not just reweight the one-to-one loss by local shift, as Differential Alignment does for alignment? Extra matches do something a scalar weight cannot. They train several distinct queries, starting from different reference points, to find the same hard object. This should help when the primary query's match flips between iterations in cluttered or foggy regions. The loss-scaling ablation tests this directly. We will also measure assignment stability: how often each pseudo-box's primary match changes between successive iterations. If loss scaling matches our accuracy and stability, the multiplicity mechanism is unnecessary.

Why not just densify one-to-one matching with Mosaic and Mixup, as in DEIM? DEIM reaches 56.5% AP on COCO with no query splitting, but its dense targets come from ground-truth labels. Stitching target images multiplies noisy pseudo-boxes uniformly, including easy background-dominated ones. DEIM also notes a remaining weakness on small objects, which are where fog hurts most. We run DEIM-style stitching on target images as an additional baseline and as a complement to our method.

Why not just attach Co-DETR's auxiliary heads? Dense anchor heads on unlabeled target images supervise the encoder over the whole image, background included. That is the regime where background alignment causes negative transfer, and Co-DETR already reports gradient conflicts from many heads. We include a single-head Co-DETR variant trained on pseudo-labels. We expect it to help overall AP less than our method on high-shift classes. If it does not, that expectation is wrong and we will say so.

Proposed Experiments

The primary benchmark is Cityscapes→Foggy Cityscapes, broken down by fog density, class, and object size. Sim10k→Cityscapes tests synthetic-to-real transfer and Cityscapes→BDD100k tests geographic transfer, with CFC-DAOD run if time allows.

The first decisive check is shift-stratified AP. We train a separate discriminator on held-out images, so the score used for evaluation is not the one used for training. We bin target test objects by that score and report AP per bin. Our claim predicts gains over uniform one-to-many concentrated in the top bins and in the person and rider classes. A flat profile would falsify the mechanism even if overall AP rises.

The second check is duplicate leakage. We measure the rate of duplicate detections on the primary group without non-maximum suppression and compare it with the one-to-one adapter. Any increase means auxiliary supervision is bleeding into the inference queries.

The third check is budget accounting. We report training memory and time relative to uniform one-to-many and verify zero inference overhead.

Finally, we track how $k_i$ evolves over training on held-out objects. This shows whether the per-object transition to one-to-one matching actually happens, or whether $k_i$ stays high on objects the student never learns. The latter would indicate that the reliability cap is not doing its job.