Is the Gap Across Domains Real? Resampling Intervals for I² in Multi-Domain ML Evaluation

When a model is tested on several hospitals, camera sites or image styles, its accuracy differs from domain to domain, and papers read those differences as evidence of distribution shift or of one method generalizing better than another. Part of that spread is only noise from finite, clustered test sets. Meta-analysis has a standard way to separate the two: the $I^2$ statistic, the share of observed spread across studies that exceeds what sampling error alone would produce. ML benchmarks almost never compute it, and when meta-analysts do, they report it as a bare number. This proposal treats each test domain as a "study" and builds confidence intervals for $I^2$ by resampling the raw test items inside each domain. Two questions become answerable with an honest error bar. Does this model's accuracy really vary across domains? Does method A's gain over method B hold across domains, or does one domain carry it?

The key opportunity is that ML evaluation, unlike a published meta-analysis, keeps item-level data. That changes what resampling can do. With only four or five domains, resampling whole domains is useless. Resampling patients, slides or images within each domain, while simulating a known amount of between-domain spread, lets us invert a test over that spread and read off an interval for $I^2$. The test needs no chi-square approximation, no normality of the metric, and no assumption that test items are independent.

Core Claim

Our central claim has two parts. First, a simulation-calibrated test inversion yields $I^2$ intervals with close to nominal coverage when there are few domains ($k$ between 4 and 10), when the true $I^2$ is at or near zero, and when the metrics are bounded, clustered and possibly non-smooth (accuracy near ceiling, worst-group accuracy, concordance index). The test inversion finds every amount of between-domain variance the data cannot reject, where each candidate is tested by an item-level resampling of the data under that variance. Second, applying it to paired model differences turns "method X improves average accuracy" into a checkable statement about whether the improvement is consistent across domains.

Write $\hat\theta_i$ for the metric (or paired difference between two models) on domain $i$. Write $v_i$ for its within-domain sampling variance, estimated by a cluster bootstrap over the domain's independent units: patients, slides, users or camera locations, not individual patches. Write $\tau^2$ for the true between-domain variance. Cochran's $Q$ is computed with weights $w_i = 1/v_i$ as in the file's definition. $I^2$ then has the standard equivalent form

$$I^2 = \frac{\tau^2}{\tau^2 + \tilde v}, \qquad \tilde v = \frac{(k-1)\sum_i w_i}{(\sum_i w_i)^2 - \sum_i w_i^2},$$

where $\tilde v$ is the "typical" within-domain variance. An interval for $\tau^2$ therefore maps one-to-one onto an interval for $I^2$.

We compute the interval for $\tau^2$ as follows. For each candidate $\tau^2$ on a grid, we generate pseudo-datasets. In each one we recenter every domain's items to a common mean, add a domain shift drawn with variance $\tau^2$, cluster-resample items, re-estimate every $v_i$, and recompute $Q$. The candidate is retained if the observed $Q$ is not in the tails of this simulated distribution. Because each $v_i$ is re-estimated in every replicate, the noise in the weights is propagated rather than ignored. Because the grid includes $\tau^2 = 0$, the interval can include zero naturally, without the ad hoc fixes that log-scale analytic intervals need when $Q \le k-1$.

Relation to Prior Work

Three papers motivate the proposal by showing that point-estimate rankings in ML evaluation hide large uncertainty. Follow the leader(board) with confidence bootstraps test items and annotator responses together to obtain p-values on a single test set. Quantifying Uncertainty in Error Consistency bootstraps trials to show that model rankings on Model-vs-Human and Brain-Score are largely noise. Both work on one pooled test set. We reuse their hierarchical resampling idea, adding training seed as a third source of variance next to items and annotator responses. We then apply it across disjoint domains, which neither paper attempts.

Two papers measure heterogeneity without quantifying its uncertainty, and so set our comparison points. Heterogeneous Datasets for Federated Survival Analysis Simulation splits survival datasets across clients with Dirichlet-controlled skew and scores heterogeneity as the fraction of client pairs with significant log-rank tests. Quantitatively Measuring and Contrastively Exploring Heterogeneity for Domain Generalization defines heterogeneity as a contrastive loss in representation space and uses it to train. Neither partitions observed spread into noise and real divergence, and neither puts an interval on its score. WILDS supplies the domains and reports large in-distribution to out-of-distribution drops. It does not ask how much of the per-domain variation survives sampling error.

SumVg constrains the design. On a non-smooth heritability sum, the delete-one jackknife badly overestimated error while parametric bootstraps did best. $I^2$ has the same kind of kink at its $\max(0,\cdot)$ truncation, so we rule out jackknife intervals and expect naive percentile bootstraps to pile up mass at exactly zero. The multiplier bootstrap analysis in Gaussian Approximation and Multiplier Bootstrap for Stochastic Gradient Descent proves non-asymptotic validity with error shrinking as $O(1/\sqrt{n})$. This suggests that the within-domain variance estimates, which rest on thousands of items, should be reliable. Coverage failures should therefore come from the small number of domains and the boundary at zero, which is exactly where the test inversion is aimed. The meta-analysis literature in the file, including Karvellas et al. and applied reviews such as the respiratory microbiota meta-analysis, illustrates the practice we want to improve: $Q$ treated as a yes/no test and $I^2$ reported without an interval.

Research Program

The work has three parts.

The first is the estimator itself. We will build the cluster-aware within-domain variance step for each metric type: accuracy, worst-group accuracy, Pearson correlation for PovertyMap, and concordance index for survival. We will also build the simulation-calibrated inversion over $\tau^2$ and a paired version in which $\hat\theta_i$ is the per-domain difference between two models evaluated on the same items. Seeds enter as an additional level. The variance of a model's domain score includes training-run variance unless the question is explicitly about a single checkpoint, and we report both versions.

The second part is a coverage study. It uses fully synthetic data with known $\tau^2$, and semi-synthetic data built two ways: by randomly splitting one real domain into pseudo-domains, which makes the true $\tau^2$ exactly zero, and by the Dirichlet splits of the survival paper, where label skew controls heterogeneity.

The third part is a reanalysis of published-style comparisons on WILDS and on the four-domain image benchmarks. For each benchmark we will report which cross-domain gaps and which method improvements have $I^2$ intervals that exclude zero.

Baselines

The naive reference point is current practice: report per-domain accuracies and their average, with no heterogeneity measure. Against that, any statement the proposed method makes about heterogeneity is new information. We will evaluate it qualitatively by counting how many claimed method gains turn out to be consistent across domains.

The strongest existing competitor is the analytic interval family from classical meta-analysis. This includes the Higgins–Thompson log-scale interval and the Q-profile inversion, which inverts the $Q$ statistic over $\tau^2$ assuming normal domain effects and known variances. The Q-profile method is the real threat because it shares our inversion logic. We will run it on the identical $\hat\theta_i$ and give it the same cluster-bootstrap $v_i$, so that the only difference is analytic versus resampled calibration. We will also run it with naive binomial variances, which is what a practitioner would plug in. The file contains no coverage figure for any analytic $I^2$ interval and no reported I² values for WILDS, PACS or the survival splits. For the benchmarks themselves the reported context is the following: WILDS gives Camelyon17 ERM accuracy falling from about 93% in-distribution to about 70% out-of-distribution; HTCL reports a 1.7% average improvement on PACS; and the survival paper reports its pairwise score rising from 0.5% to 94.0% on SUPPORT as label skew increases. Beating the analytic baseline means achieving coverage within two Monte Carlo standard errors of 95% wherever the analytic interval falls outside that band, especially at $k \le 5$ and true $I^2$ near zero, while producing intervals no wider than necessary.

We also run a percentile bootstrap that resamples items within domains and recomputes $I^2$ directly. This isolates our key ingredient, the test inversion over $\tau^2$ with simulated domain shifts, from resampling as such. If this simpler bootstrap matches our coverage, the inversion is not needed. We expect it to fail at the zero boundary, because resampling items under the observed data cannot produce the spread of $Q$ under small true $\tau^2$.

A second ablation drops cluster resampling in favor of independent patch-level resampling. This tests whether clustering, rather than the inversion, explains most of the gain.

All methods share the same item-level predictions, the same seeds and the same number of resampling draws, matched in compute.

Why not just…?

Why not just report Cochran's Q p-value? $Q$ tells you whether there is detectable spread, not how much. Its power is tied to test-set size and number of domains, as Karvellas et al. note. On WILDS-sized test sets, trivial gaps become significant, while with four PACS domains real gaps can be missed. Our interval reports magnitude with uncertainty. The pseudo-domain experiment will show directly whether $Q$ p-values with naive variances reject far more often than 5% when the truth is zero.

Why not just use the analytic Q-profile interval? This is a serious alternative. When $k$ is moderate and the plug-in variances are accurate, it may match us at a fraction of the cost. Our bet is that it fails in two specific places. It treats $v_i$ as known, although each is estimated from clustered data. It assumes normal domain effects for metrics bounded near 1. If the coverage study shows Q-profile with cluster-bootstrap variances is already nominal, we will recommend that cheaper combination and report the within-domain variance step as the actual contribution.

Why not just resample domains? With four or five domains, a domain-level bootstrap has only a handful of distinct resamples and cannot represent a sampling distribution. Domain-level resampling is also not what $I^2$ asks about. $I^2$ concerns the variance at the item level within domains compared with the variance across domains. We include it in the coverage study only to document this failure.

Why not count significant pairwise tests, as in the federated survival paper? The pairwise score of Heterogeneous Datasets for Federated Survival Analysis Simulation grows with test power and with the number of pairs, and it says nothing about what fraction of spread is real. On the same SUPPORT splits we will compute both quantities across the Dirichlet $\alpha$ grid. We will check whether the $I^2$ interval tracks the controlled skew more smoothly and separates quantity skew from label skew. Under quantity skew, true divergence should be near zero even though per-client noise changes.

Proposed Experiments

The first decisive check is the null calibration. We will split single WILDS domains, such as one Camelyon17 hospital or one iWildCam location group, and single PACS domains into 4 to 10 random pseudo-domains of realistic size. Because true $\tau^2$ is zero, a valid method's interval should include zero about 95% of the time, and its point estimate should not drift upward with test-set size. We expect patch-level variances in Camelyon17 to make every naive method report high heterogeneity among patches from the same hospital. That would be a concrete demonstration of why cluster resampling matters.

The second check is the power and coverage curve. Using synthetic shifts injected into the pseudo-domains, and the Dirichlet label-skew splits of SUPPORT, GBSG2 and METABRIC with concordance index as the metric, we will trace coverage and interval width against true $I^2$ for every baseline.

The third is the reanalysis. For ERM and robust training methods on Camelyon17, PovertyMap, FMoW, iWildCam and CivilComments groups, and for domain generalization methods on PACS, OfficeHome, VLCS and TerraIncognita, we will report the interval for $I^2$ on raw per-domain performance. We will also report the interval on paired improvements, each with and without seed variance.

The result that would change practice is a demonstration that some reported average gains on four-domain benchmarks have paired $I^2$ intervals consistent with a uniform improvement, while others are concentrated in one domain. The result that would sink the proposal is equal coverage from the analytic Q-profile interval fed with cluster-bootstrap variances, and we will report that outcome plainly if it occurs.