Two-Stage Bootstrap Intervals for Choosing Domain-Adapted Models Without Target Labels

A model trained on labeled data from one setting (the source, such as product photos) is often deployed in another (the target, such as webcam images) where no labels exist. This is unsupervised domain adaptation. Its quiet failure point is the last step: choosing which trained model, hyperparameter setting, or checkpoint to ship. Without target labels, practitioners rank candidates with a surrogate score, such as accuracy against pseudo-labels produced by an auxiliary model, and pick the top one. That score is a single number. Nobody asks whether the gap between the first- and second-ranked candidates is larger than the noise in the score itself. Behavioral scientists who test mediation effects, meaning effects that pass through an intermediate variable, learned long ago not to trust point estimates of compound quantities. They resample the data thousands of times and read significance off the empirical distribution.

This note proposes the same discipline for model selection, with one twist that matters. A surrogate score is a two-stage estimator. An auxiliary model is fitted on source data, and its outputs are then averaged over target data. Most of the error comes from the first stage. A bootstrap that resamples only target points would miss that error and produce intervals that are far too narrow. We resample source and target jointly, refit the cheap auxiliary model on every resample, and bootstrap paired differences between candidates rather than their raw scores. This yields a set of candidates that cannot be statistically distinguished from the apparent best. We then choose within that set using ensemble agreement. The result is a selection rule that knows when it is guessing.

Core Claim

Surrogate target-risk estimates for unsupervised model selection behave like the indirect effect $a \times b$ in mediation analysis. They compound a fitted first-stage quantity with a second-stage average, and their sampling distribution is skewed and dominated by the first stage. A percentile bootstrap that refits the first stage gives intervals for pairwise risk differences that actually cover the true differences. Restricting the final choice to the candidates whose intervals overlap the best one reduces both average regret and catastrophic selections, compared with ranking on point estimates. Because the auxiliary model is a linear head on frozen features, 1,000 refits cost seconds per task. This sidesteps the computational objection that usually rules out bootstrapping deep models.

Grounding in Prior Work

The resampling method comes from the mediation literature. Bootstrapping for Indirect Effects and Mediation Analysis and applied studies such as the perceived-eeriness study of AI-generated ads establish two points. First, products of estimated coefficients have skewed, heavy-tailed sampling distributions. Second, symmetric normal-theory intervals of the Sobel type lose power and misstate error rates, while percentile bootstrap intervals from about 5,000 refits do not. We borrow the percentile construction and the insistence on refitting every stage. We do not borrow the i.i.d. single-sample assumption, which breaks here because the data come from two distributions.

Several existing selection methods supply the surrogate scores we wrap. Pseudo-Label Validation splits the source data, trains an imputation model to produce soft pseudo-labels on the target, and scores candidates against those labels. It proves that imputation variance averages out over the target set while imputation bias does not, and so recommends undersmoothing the imputer. Our design follows directly from that asymmetry. The error that survives large target sets lives in the imputer, so an interval has to resample what the imputer was trained on.

Deep Embedded Validation estimates feature-space density ratios with a logistic-regression domain classifier and reduces variance with control variates. It is another two-stage estimator, but it reports only a point estimate. IW-GAE estimates accuracy within confidence groups by fitting importance weights so that they reproduce the known source accuracy. Its authors report non-identifiability: different weight solutions fit the source equally well yet disagree on the target. That is precisely the kind of first-stage instability a refit bootstrap exposes.

Two papers motivate the proposal by showing that point ranking fails, and they also constrain it. EnsV shows across 131 tasks that entropy- and neighborhood-based scores select the worst candidate up to 80 times. Its own fix scores each candidate by agreement with the ensemble average of all candidates. It made zero worst-case selections, but it degrades when most of the pool is mediocre, and it cannot resolve narrow margins. MixVal probes decision boundaries with mixup samples. On DomainNet it comes within a point of the best possible choice, but it reports no variability. Our rule uses EnsV-style agreement as the tiebreaker, but only inside a statistically trimmed pool. The trimming is designed to remove EnsV's mediocre-ensemble failure.

Finally, UBR²S also uses "resampling" in domain adaptation. It samples pseudo-labels from Gaussians fitted to dropout outputs during training. That is a training heuristic with a parametric shape assumption, not inference about a selection score, so it is orthogonal to this proposal.

Research Program

Let $k = 1, \dots, m$ index candidate models. Let $\hat{R}_k(S, T)$ be a surrogate target risk computed from source sample $S$ and target sample $T$; for pseudo-label validation, this means fitting the imputer on part of $S$ and scoring candidate $k$ against the imputer's soft labels on $T$. Let $\hat{k}$ be the candidate with the lowest point estimate. In each bootstrap round $b$, we draw source and target resamples $S^*_b$ and $T^*_b$ independently with replacement and refit the auxiliary model on $S^*_b$. We then record

$$\Delta^*_{k,b} = \hat{R}_k(S^*_b, T^*_b) - \hat{R}_{\hat{k}}(S^*_b, T^*_b),$$

the bootstrap difference in surrogate risk between candidate $k$ and the apparent best. Both terms use the same refitted imputer, so imputer errors shared by the two candidates cancel. Only the part of the error that changes the ranking remains, which makes intervals on differences much tighter than intervals on raw risks.

The plausible set $\mathcal{P}$ contains every candidate whose lower 5th percentile of $\Delta^*_{k,\cdot}$ is at or below zero, meaning it is not significantly worse than $\hat{k}$. The 5th percentile corresponds to a one-sided test at the 5% level. The final choice is the member of $\mathcal{P}$ that agrees most with the average prediction of the members of $\mathcal{P}$. When $\mathcal{P}$ contains a single candidate, the rule reduces to the point argmin.

We will build this wrapper for three first-stage estimators. These are the pseudo-label imputer, the deep embedded validation domain classifier, and the importance-weighted group accuracy estimator. All three are refitted as heads on frozen candidate features. If the source and target samples contain known structure, such as hospitals in Camelyon17 or images from the same object instance, we resample whole clusters rather than individual points.

We will also study the method analytically. The question is when shared-imputer cancellation makes paired-difference intervals valid, and how the interval width scales with the imputer's undersmoothing level. On the second question, theory and practice may pull in opposite directions. Undersmoothing reduces bias but should widen the intervals, and characterizing that tradeoff is itself a result.

We state one limitation up front. No resampling scheme can detect bias that is common to every resample, such as an imputer that is systematically wrong under extreme shift. We therefore measure coverage of the true pairwise target-accuracy differences and report where it fails, rather than claiming coverage of absolute target risk.

Baselines

The naive reference point is source validation: pick the candidate with the best held-out source accuracy. Its failure is documented in Pseudo-Label Validation. On Camelyon17 it chose an overfit epoch-8 checkpoint at 53% target accuracy, while pseudo-label validation chose epoch 1 at 73%, matching the oracle. On Raisin, source validation reached target risk 0.502, against 0.428 for pseudo-label validation and 0.373 for the oracle. Any new method must clear this baseline easily.

The strongest existing competitor for avoiding disasters is EnsV, which reported 0 worst-case selections across 131 closed-set tasks. The file gives no average-accuracy figure for EnsV, so we will rerun it ourselves. It is the strongest competitor because it already achieves the robustness goal we claim. On the same candidate pools, our method must match it on worst-case selections, show lower mean regret, and especially beat it on pools deliberately constructed so that most candidates are poor.

For accuracy near the oracle, MixVal is the strongest reported method. On DomainNet closed-set tasks it reached 69.68% average selected accuracy, against an oracle best of 70.34%, entropy at 64.34%, and SND at 63.05%. On a ViT-based DomainNet task it reached 54.78%, against 28.21% for entropy. The remaining headroom there is about 0.7 points, so beating MixVal on DomainNet would require closing a meaningful fraction of that gap. Any improvement must be shown across random seeds, not as a single-run number.

Deep embedded validation reported selections "nearly identical" to oracle selection on Office-31 and VisDA, but the file gives no reported number for its selected accuracies. We will rerun it on identical candidate pools.

All selectors will see the same candidate checkpoints, the same source split, and the same frozen features. Each auxiliary model receives the same hyperparameter budget, chosen on source data only. All methods are run over five seeds of the candidate-generation sweep.

Three ablations isolate the key ingredient. The first bootstraps only the target set, with no imputer refitting. The second bags the imputer over the same resamples but takes the point argmin of the bagged score, with no plausible set. The third applies EnsV to the plausible set built from unpaired, rather than paired, intervals.

Why not just…?

Why not just bootstrap the target examples, which is the obvious resampling? With thousands of target points, target-only resampling captures only averaging noise. The pseudo-label validation analysis says that noise is the negligible part, and that the error that persists sits in the imputer. We predict target-only intervals will badly undercover the true pairwise differences. This is the first ablation. If it covers as well as the two-stage version, the central mechanism is wrong, and we will report that.

Why not just bag the imputer and keep point ranking? This is a real threat. Averaging imputers refitted on bootstrap resamples may by itself fix most selection errors, in which case the gain would come from variance reduction, not from intervals. The second ablation tests this directly. The proposal is justified only if the plausible-set rule beats bagged argmin on worst-case and narrow-margin tasks. If bagging matches it, the honest contribution narrows to calibrated uncertainty reports, and we would say so.

Why not just use EnsV, which already never picks the worst model? EnsV's own analysis concedes two weaknesses: it fails when the candidate pool is mostly mediocre, and it cannot separate close top candidates. Trimming the pool first targets the first weakness directly. Our adversarial-pool experiment, in which we inject 50 to 80% badly tuned candidates, is designed to show whether EnsV degrades while the trimmed version does not.

Why not just use a normal-approximation standard error on the score? The mediation literature shows that symmetric intervals misstate error for compound estimators. We will check skewness of the bootstrap difference distributions and report how often a normal interval and a percentile interval disagree about membership in the plausible set.

Proposed Experiments

The candidate pools are hyperparameter and checkpoint sweeps of Deep CORAL, ADDA, and a self-training method on Office-31, Office-Home, VisDA-2017, and DomainNet-126, using the 131-task closed-set suite from EnsV wherever it can be reconstructed. We add Camelyon17 with a DINOv2 backbone, following pseudo-label validation, and Raisin for kernel-model checks where the theory is tight.

The primary metrics are three. The first is regret, the oracle-best target accuracy minus the selected candidate's accuracy. The second is the count of worst-case selections. The third is empirical coverage of the 95% intervals for true pairwise target-accuracy differences.

The decisive check is coverage. On each task we compute true differences using held-out target labels, which are used only for evaluation. The method succeeds if two-stage paired intervals reach near-nominal coverage where target-only intervals fall well short. We will also measure how plausible-set size grows as the true margins shrink, which shows whether the method correctly signals "too close to call." A second check plots regret against pool contamination to test the mediocre-ensemble claim.

A third check asks whether intervals are informative in practice. Narrow plausible sets should predict low regret. Tasks with wide plausible sets should be exactly those on which every point-estimate method shows high seed-to-seed variance.