Object detectors break when moved to a new environment, such as clear weather to fog or rendered game footage to real streets. The standard fix, domain adaptation, pulls the detector's internal features for the two environments toward each other. The better versions do this only on "foreground," the regions that contain objects, because skies and road textures differ wildly between environments and aligning them pollutes the features. These methods share a weakness. The component that decides what counts as foreground is the detector's own proposal module, and that module was trained on source-domain categories. In fog it misses faint objects. It also treats any object from a category it has never seen as background. Foreground alignment then leaves out exactly the objects that most need it, and the proposal module gets worse at finding them. The loop feeds itself.
This proposal separates two questions: is there an object here and what is it. They get two different alignment rules. We build on a detector whose region features are split into a length and a direction, where the length encodes category-free "objectness" and the direction encodes category, as in Exploring Orthogonality in Open World Object Detection. We align the distribution of objectness lengths across domains using boxes placed at random. Because those boxes are drawn independently of the detector, their sample is unaffected by what the detector currently believes. We align category directions only on regions that are confidently recognized. Every object-like region, including unknown classes, gets its objectness repaired. Only regions the model can name have their semantics pulled toward source classes. The prediction is that this recovers target recall for both faint known objects and novel-class objects, where current confidence-weighted alignment either ignores them or collapses them into known classes.
Core Claim
In adaptive detection where the target has both an environmental shift and unseen categories, two things should drive the benefit on unknown-class recall. The first is the choice of sample for objectness alignment: random boxes rather than the model's own proposals. The second is keeping objectness alignment apart from category alignment. Aligning a one-dimensional objectness statistic over random boxes needs no foreground mask. That removes the proposal-quality feedback loop that Collaborative Training Between Region Proposal Localization and Classification identified as the main bottleneck. Restricting category alignment to confident known regions prevents the collapse of novel objects into known clusters that Novel Scenes & Classes documents. If the claim is right, the gain should be largest on novel-class recall and on low-contrast known objects, and it should shrink or vanish if the random boxes are replaced by the detector's own proposals.
Relation to Prior Work
The closest existing work already uses class-agnostic signals to decide where to align, so the obvious combination is taken. Three papers weight or mask alignment with the model's own objectness estimate, and differ only in how they obtain it.
- Collaborative Training weights alignment by the confidence of the region proposal network (RPN), the module that suggests candidate boxes.
- Every Pixel Matters weights adversarial alignment by predicted pixel-wise objectness and centerness maps in an anchor-free detector.
- O²net masks backbone alignment with class-agnostic target pseudo-labels.
All three draw the foreground estimate from the model being adapted, so their masks inherit its target-domain failures. Gong et al. acknowledge that alignment quality stays tied to initial pseudo-label quality. All three also assume the two domains share one category set. We do not use a model-derived mask for the objectness alignment at all.
D-adapt splits category and box adaptation into separate adaptors and retrains through three pseudo-label rounds. It motivates our separation, since it shows that adversarial alignment on backbones harms foreground/background discrimination. Its adaptors, however, are closed-set and confidence-weighted. SOMA is the direct competitor on adaptive open-set detection. It uses triangle graph motifs on Deformable DETR queries so that novel objects stay "something." It does not isolate objectness as a separate quantity to align, and it is restricted to the last decoder layer for cost reasons.
Two papers supply the method. Exploring Orthogonality gives the length/direction decomposition, trained with a decorrelation loss, but only within a single domain. A generic domain discriminator on the whole feature vector gives no guarantee of keeping that separation; the research file raises this concern but does not demonstrate it, and our ablations test it directly. Random Boxes Are Open-world Object Detectors shows that random Gaussian boxes, used in place of learned proposals during training, remove category bias. We reuse that idea for a new purpose: as a sample for measuring domain statistics that the detector cannot select.
Two papers make the motivating point that target object localization is the real gap, but both need target boxes. ODAPT fine-tunes a class-agnostic detector on about 100 labeled target frames. ABRA relabels target boxes as generic "object." Our setting uses no target boxes.
Research Program
The base detector computes each region feature as $f = m\,u$, where $m = \lVert f \rVert$ is the objectness length and $u$ is a unit direction used for classification. We train this with the decorrelation loss of Sun et al. on labeled source data.
At every adaptation step we do the following:
- Sample $N$ random boxes in a source image and $N$ in a target image, following RandBox's Gaussian box distribution.
- Pool their features and compute the lengths.
- Sort the lengths in each domain, giving $m^s_{(1)} \le \dots \le m^s_{(N)}$ for source and $m^t_{(1)} \le \dots \le m^t_{(N)}$ for target.
- Minimize the one-dimensional optimal-transport distance between them:
$$ \mathcal{L}_{\text{mag}} = \frac{1}{N}\sum_{i=1}^{N} \big( \mathrm{sg}(m^s_{(i)}) - m^t_{(i)} \big)^2 , $$
where $\mathrm{sg}$ stops gradients so that only target features move. In one dimension, sorting solves the optimal transport problem exactly. This loss therefore has no discriminator and no mask. It touches only the objectness length and leaves direction alone.
The main risk is that the fraction of boxes covering objects differs between domains, for example sparse clipart versus dense street scenes. Matching the full distribution would then push background lengths up. We therefore also test matching only the upper quantile range, $i > (1-q)N$, and treat $q$ as the key hyperparameter.
Category alignment uses a standard adversarial discriminator, applied only to the direction $u$ and only on proposals whose top known-class probability exceeds a threshold. Target regions with high $m$ but low known-class confidence get no direction alignment. These are our candidate unknowns. They benefit from $\mathcal{L}_{\text{mag}}$ but are never pushed toward a source class.
Beyond accuracy, we will build a diagnostic that tracks the feedback loop over training. Using held-out target ground truth for evaluation only, we measure the fraction of target objects whose objectness falls below the detection threshold, split into known classes, novel classes, and a low-contrast subset (the objects with the lowest local contrast).
Baselines
The naive reference point is the source-only polar detector with no adaptation. Every Pixel Matters reports 25.6% mAP for its own non-adapted baseline on Cityscapes→Foggy Cityscapes. We will re-measure the number for our detector rather than borrow it.
For closed-set adaptation, the strongest foreground-aware competitors are:
- O²net: 46.8 mAP on Cityscapes→Foggy Cityscapes and 54.1 car AP on Sim10k→Cityscapes.
- Every Pixel Matters: 40.2% mAP on Cityscapes→Foggy Cityscapes with ResNet-101.
- D-adapt: 49.0 mAP on VOC→Clipart1k.
These numbers come from different detectors, so direct comparison would mostly reflect architecture. We will instead re-implement the three weighting rules on our own backbone and detector with identical schedules and an equal hyperparameter search budget: RPN-confidence weighting as in Collaborative Training, centerness weighting, and pseudo-label masking. We will report published numbers alongside for context. Beating them means matching their closed-set mAP within about one point while clearly improving low-contrast recall.
For the open-set setting, the strongest method is SOMA. On Cityscapes→Foggy Cityscapes it reports base-class mAP of 50.87% and novel-class recall of up to 15.67%. We will run our method on Deformable DETR on SOMA's splits. Our objectness head is added to the decoder queries, which keeps the architecture matched. Success means novel recall clearly above 15.67% without falling below SOMA's base mAP.
Three ablations isolate the key ingredient:
- Proposals instead of random boxes. Compute $\mathcal{L}_{\text{mag}}$ over the detector's own top-scoring proposals. This tests whether escaping the model's own selection is what matters.
- Confidence gate replaced by objectness gate. Gate direction alignment by objectness rather than known-class confidence. This lets unknowns be aligned toward known classes, and tests the separation.
- Whole-feature discriminator. Apply an ordinary discriminator to the entire feature $f$. This tests whether mixing length and direction during alignment hurts.
Why not just…?
Why not just use SAM or Grounding DINO to supply target foreground masks? This is a real threat. A frozen foundation model is also independent of the adapted detector, so it too could break the feedback loop. The evidence that it falls short is indirect. ABRA reports zero-shot Grounding DINO at 27.7 mAP on Foggy Cityscapes and 9.2 mAP on SDGOD's Night-Rainy split. However, those figures measure class-labeled detection, not category-free mask recall. We will therefore run SAM-masked foreground alignment as a full baseline. If it matches our method on novel recall, our contribution shrinks to separating length from direction, and we will say so. Our method still differs in needing no large external model at training time.
Why not just train a class-agnostic RPN and weight alignment by its score? This is essentially Collaborative Training's RPN weighting. Sun et al. and RandBox both report that binary-trained proposal modules still suppress unannotated objects. Unknown recall is 8.4% for the baseline in Sun et al., and 4.9–7.5% for ORE and OW-DETR in RandBox's comparison. The proposals-instead-of-random-boxes ablation tests this failure directly within domain adaptation.
Why not just apply RandBox on the source and skip adaptation? RandBox removes category bias but has no mechanism for environmental shift. It reports 10.6% unknown recall on VOC/COCO within a single domain; for adverse-domain unknown recall, no reported result was found on Lacuna for RandBox. Running source-trained RandBox on the Foggy open-set split answers this. It also costs about 10,000 boxes per image at inference, whereas we use random boxes only during training.
Why not just do global adversarial alignment on the polar features? Every Pixel Matters and D-adapt already show global alignment losing to foreground-aware alignment in closed-set adaptation. The whole-feature discriminator ablation checks whether that result still holds once the features are split into length and direction.
Proposed Experiments
Benchmarks. We use Cityscapes→Foggy Cityscapes in both its closed-set form and SOMA's open-set split, plus Sim10k→Cityscapes, and VOC→Clipart1k to stress the object-density assumption. We add SDGOD Night-Rainy as a severe-contrast test.
Metrics. We report mAP or car AP, base-class mAP, novel-class recall, class-agnostic average recall on target, and the per-epoch missed-object diagnostic described above.
Decisive checks. The claim survives only if four things hold:
- Random-box magnitude matching beats proposal-based magnitude matching on novel recall and on low-contrast recall. This check matters most.
- The missed-object fraction for novel classes falls during our training, while it stays flat or rises under confidence-weighted alignment.
- The upper-quantile variant of $\mathcal{L}_{\text{mag}}$ does not inflate background false positives on Clipart1k. We measure this by false positives per image at a fixed recall.
- Our method at least matches SAM-masked alignment on novel recall.
If check 1 fails, the random-box idea is not doing the work. If check 4 fails, the foundation-model route is the better answer, and the paper becomes a negative result about learned objectness under shift.
References
- Exploring Orthogonality in Open World Object Detection
- Collaborative Training Between Region Proposal Localization and Classification for Domain Adaptive Object Detection
- Novel Scenes & Classes: Towards Adaptive Open-set Object Detection
- Every Pixel Matters: Center-Aware Feature Alignment for Domain Adaptive Object Detector
- Improving Transferability for Domain Adaptive Detection Transformers
- Decoupled Adaptation for Cross-Domain Object Detection
- Random Boxes Are Open-world Object Detectors
- Object-based (yet Class-agnostic) Video Domain Adaptation
- ABRA: Teleporting Fine-Tuned Knowledge Across Domains for Open-Vocabulary Object Detection