Which Cheap Trials Can Be Trusted When Tuning Domain-Adaptive Detectors

Tuning a detector that has to work on a new visual domain is expensive. Examples include a car detector trained on synthetic scenes and deployed on real streets, or a tree-crown detector trained on one rainforest site and deployed on another. Every trial is a full multi-stage adaptation run on high-resolution images. The standard way to save compute is to run most trials cheaply first: at lower image resolution, on fewer images, or for fewer iterations. The expensive setting is reserved for configurations that already look good. This trick has been shown to work for image classification. For cross-domain detection there are good reasons to expect it to mislead. Shrinking an image erases small objects. Early in adaptation, the model's self-generated labels are known to be badly wrong. And the small labeled sample of the target domain used to score trials is itself noisy.

We propose to measure, for each hyperparameter separately, the cheapest setting at which its ranking of candidate values still agrees with the expensive setting. We then build a cost-first search that tunes each hyperparameter only at a setting cheap enough to save compute and faithful enough to trust. Put simply, the claim is that learning rate may be safely tuned on tiny, blurry, short runs, while the pseudo-label confidence threshold and anchor sizes may not. A search that knows the difference should find better adapted detectors per GPU-hour than either uniformly cheap or uniformly expensive search.

Core Claim

Cost-prioritized search starts in the cheapest region of the configuration space and expands outward only when cheaper settings are beaten. Two methods in this family supply our search mechanism. Frugal Optimization for Cost-related Hyperparameters starts a randomized local search at a low-cost configuration. ECONOMIC HYPERPARAMETER OPTIMIZATION WITH BLENDED SEARCH STRATEGY (BlendSearch) mixes global and local search inside an "admissible region" that begins near cheap configurations and grows over time. Resolution is the natural cost axis for vision. Speeding up the Hyperparameter Optimization of Deep Convolutional Neural Networks showed with functional ANOVA that the importance of learning rate, batch size and regularization is stable across input resolutions for classification. The same paper also found that structural choices such as depth are not stable, because pooling shrinks small inputs too early.

That caveat both motivates this work and constrains it. Transfer across fidelities is a property of individual hyperparameters, not of the whole search. We claim three things about cross-domain detection. First, ranking agreement between cheap and full settings splits hyperparameters into distinct groups. Optimizer parameters should transfer across resolution. Hyperparameters that interact with object scale or with pseudo-label quality should not. Second, the right cheap axis differs per group. Resolution may be harmless for adversarial loss weight but damaging for anchor scales. Truncated adaptation may be harmless for learning rate but damaging for the pseudo-label threshold. Third, a search that picks a fidelity per hyperparameter, using agreement estimates gathered cheaply during the search, beats both plain BlendSearch and standard multi-fidelity bandits on best target accuracy found per unit of compute.

To make this precise, write $\rho_j(f)$ for the Kendall rank correlation between the target-domain scores of candidate values of hyperparameter $j$ evaluated at cheap fidelity $f$ and at full fidelity, with the other hyperparameters held at their incumbent values. Here $f$ indexes a combination of resolution scale, target-subset fraction, adaptation length and validation-set size. The search tunes $j$ at

$$f_j^\star = \arg\min_{f} \; c(f) \quad \text{subject to} \quad \rho_j(f) \ge \tau,$$

where $c(f)$ is the GPU-hour cost of one trial at fidelity $f$ and $\tau$ is a trust threshold that we sweep in the experiments.

Why This Is Not Already Done

Adjacent work leaves a specific opening. Tune As You Scale: Hyperparameter Optimization For Compute Efficient Training (CARBS) models performance and cost jointly with Gaussian processes to trace compute Pareto frontiers. It was validated only on scalar-objective tasks: reinforcement learning, language modeling and Imagenette. It never faces domain shift or localization. BOHB: Robust and Efficient Hyperparameter Optimization at Scale and Efficient Hyperparameter Optimization with Adaptive Fidelity Identification (FastBO) assume that early or subsampled evaluations rank configurations faithfully. FastBO relaxes this by fitting learning curves to choose a stopping point per configuration. Neither checks faithfulness separately for each hyperparameter. DADA: Differentiable Automatic Data Augmentation searches cheaply on ImageNet classification and transfers the fixed result to COCO detection. The search never runs on the structured task.

On the detection side, Cross-Domain Adaptive Teacher for Object Detection names automated setting of the confidence threshold as an open limitation. Decoupled Adaptation for Cross-Domain Object Detection (D-adapt) adds a three-round cascade that multiplies training cost. Neither paper uses any cost-aware tuning. The closest single work is the resolution-scaling study, and it differs from ours in three ways. It covers classification only. It treats resolution as the only cheap axis. And it partitions hyperparameters after the fact, rather than using that information to steer a search under domain shift.

Research Program

The work has two stages. The first is a measurement study, and it has standalone value even if the search method fails. We take Adaptive Teacher and D-adapt as the adapted detectors, keeping their released backbones and training recipes. For each we define a search space of roughly eight hyperparameters. The first group is optimizer-level: learning rate, weight decay and warmup. The second is adaptation-level: adversarial loss weight, pseudo-label confidence threshold, teacher exponential-moving-average rate, and the ratio of weak to strong augmentation. The third is scale-level: anchor scales and the minimum box size kept in training. The cheap axes are four:

  • resolution scale (0.25, 0.5, 1.0 of native);
  • fraction of unlabeled target images used (10%, 30%, 100%);
  • adaptation length as a fraction of the published schedule;
  • size of the labeled target validation set used for scoring.

That last axis brings in the dataset-scale question in a form specific to adaptation. Tuning needs some labeled target images. How many are needed before rankings stabilize is itself unknown.

For a sampled set of configurations around the published defaults, we run full-fidelity trials and a grid of cheap trials, then estimate $\rho_j(f)$ with bootstrap intervals. We also break target mean average precision down by object-size bucket. The breakdown should show why rankings flip at low resolution. Our hypothesis is that low-resolution trials effectively optimize large-object accuracy.

The second stage builds the search. We extend BlendSearch's admissible region so that it grows along fidelity axes as well as cost-related hyperparameters. The search keeps a running estimate of $\rho_j(f)$ from occasional paired "audit" trials, which evaluate the same pair of configurations at two fidelities. It promotes hyperparameter $j$ to a more expensive fidelity when its estimated agreement drops below $\tau$. Audit trials are charged to the search budget, so the search pays for its own trust estimates. This also answers BlendSearch's known need for a human to say which dimensions drive cost: the audits discover which dimensions drive fidelity error.

Baselines

The naive reference point is the published default configuration of each detector, trained once at full fidelity. For this reference the file gives reported numbers. Adaptive Teacher reports 50.9% mAP on Cityscapes→Foggy Cityscapes, 49.3% mAP on VOC→Clipart1k and 59.9% mAP on VOC→Watercolor2k (Cross-Domain Adaptive Teacher for Object Detection). D-adapt reports 49.0 mAP on VOC→Clipart1k, 40.5 mAP on VOC→Comic2k and 50.3 mAP on Sim10k→Cityscapes (Decoupled Adaptation for Cross-Domain Object Detection). We will reproduce these under our protocol before comparing, because any tuning gain must be measured against our reproduction, not against the paper's number. These defaults are a hard target. They were likely chosen with knowledge of target results, so failing to beat them with an honest small validation set would itself be informative.

The strongest existing search competitor is BlendSearch run as published, with resolution and training length declared as cost-related dimensions. It is the strongest because it is the cost-first method with the best reported compute efficiency in the file. On a production Turing-NLRv2 fine-tuning task it reached an F1 of 0.652 in 12 GPU-hours, against 0.640 for ASHA-16 in 384 GPU-hours (ECONOMIC HYPERPARAMETER OPTIMIZATION WITH BLENDED SEARCH STRATEGY). No reported result was found on Lacuna for BlendSearch, BOHB or FastBO on cross-domain object detection. The multi-fidelity competitors are BOHB with training length as the budget and FastBO with its adaptive stopping point. We also run a resolution-staged search in the style of Hinz et al., using the Tree-structured Parzen Estimator.

Every method receives the same search space, the same labeled target validation set, the same pool of unlabeled target images, and the same total GPU-hour budget. Budgets are expressed as multiples (4, 8 and 16) of one full-fidelity adaptation run. Each method gets five seeds. The final chosen configuration of every method is retrained at full fidelity and scored on a held-out target test split never used in search.

Beating BlendSearch means a higher final held-out target mAP at the same budget, with non-overlapping seed intervals, or reaching BlendSearch's final mAP with at least a third less budget. Beating the default means a gain over our reproduced default that exceeds seed variation.

Two ablations isolate the key ingredient. The first uses a fixed per-hyperparameter fidelity assignment taken from the offline Stage-1 study, with no online audits. It tests whether online estimation matters or whether a lookup table suffices. The second applies uniform fidelity: every hyperparameter is tuned at the fidelity that is optimal on average. It tests whether per-hyperparameter gating matters at all.

Why not just…?

Why not just use the published defaults? Doing nothing costs zero search compute, and the published numbers are strong. The case for tuning rests on two facts. Adaptive Teacher's authors themselves name threshold selection as unsolved. And defaults tuned on Cityscapes-style shifts need not suit a new shift such as cross-site forest imagery. This alternative is a real threat on the classic benchmarks. The experiment settles it by reporting gain over the reproduced default per benchmark. We expect the gain to be modest on Foggy Cityscapes and larger on SelvaBox, where no default was ever tuned.

Why not just run BOHB or ASHA with fewer epochs as the cheap setting? Bandit schedulers assume early scores predict final scores. In teacher-student adaptation, early pseudo-labels carry the false positives that Adaptive Teacher reports cutting by up to 35%, so early rankings of the threshold and EMA rate should be unreliable. FastBO's own finding, that crossing learning curves hurt fixed schedules, points the same way. Our Stage-1 measurement of $\rho_j$ against adaptation length tests this directly. If truncated runs rank all hyperparameters faithfully, BOHB will match us, and we will say so.

Why not just tune at low resolution, as was done for classification? The resolution study itself found that structural hyperparameters depend on input size. Detection adds a mechanism classification lacks: small boxes vanish below the resolution the feature pyramid can see. The size-bucketed accuracy breakdown will show whether low-resolution winners lose on small objects at full scale. The uniform-fidelity ablation shows what that costs in final mAP.

Why not just do the offline study once and hand-assign fidelities? This is the most plausible competitor, and it may well suffice if fidelity agreement transfers across benchmarks. The fixed-assignment ablation tests exactly this. We derive the assignment on VOC→Clipart1k and apply it unchanged to Sim10k→Cityscapes and SelvaBox. If it matches the online version, the contribution becomes the measurement study plus a reusable assignment, which is simpler and still useful.

Proposed Experiments

The benchmarks are Cityscapes→Foggy Cityscapes, VOC→Clipart1k, VOC→Watercolor2k and Sim10k→Cityscapes, as used by the two adaptation papers. To test scale and small objects under genuine site shift, we add CanopyRS/SelvaBox. It contains 2,400 high-resolution images with 83,000 tree-crown boxes from Brazil, Ecuador and Panama. We define leave-one-country-out adaptation splits, which make resolution-dependent fidelity error acute.

The primary metrics are:

  • held-out target mAP of the final configuration;
  • anytime curves of best validation mAP against GPU-hours, summarized by area under the curve;
  • $\rho_j(f)$ with confidence intervals for every hyperparameter and fidelity;
  • mAP split by object size.

Four results would be decisive. The first check is whether $\rho_j(f)$ actually separates the hyperparameter groups we hypothesize. If all hyperparameters behave alike, per-hyperparameter gating has no reason to exist, and the study reports that instead. The second is whether the gated search beats plain BlendSearch at matched budget on at least three of the five benchmarks, with the largest margin on SelvaBox. The third is how the audit overhead scales. If audits consume most of the budget at small budgets, the method is only worthwhile above a budget threshold we will report. The fourth is how many labeled target validation images are needed before $\rho_j$ at full resolution stabilizes. This answers the dataset-scale question in its most practical form: how much target labeling makes tuning meaningful at all.