Geographic Transfer Failure Is an Interaction Effect: A Factorial Test on Commuting-Flow Models

Models trained on data from one place often break when used somewhere else, and the standard explanation is that the new place "looks different." Current audits measure that difference as a few summary numbers, such as how different the input features are or how differently they cluster in space. They then relate those numbers to transfer accuracy with a linear regression over whatever region pairs happen to exist. This proposal argues that the explanation is missing its most important part. A model fails in a new region when the combinations of features it relies on stop co-occurring the way they did at home, even if each feature on its own looks familiar. Observational audits cannot detect this, for two reasons. The shift measures they use tend to move together across real regions, and the models' reliance on feature combinations is never measured.

We propose to test this claim with a controlled factorial design on the CommutingODGen commuting-flow data. A factorial design crosses several factors at chosen levels so that their separate and joint effects can be estimated. Here we select source–target region pairs so that the known shift components vary independently of one another, rather than accepting them as they occur. We then add a new predictor of transfer failure: shift weighted by how much the model's predictions depend on pairwise feature interactions, measured with interaction-aware Shapley attribution. The practical payoff is a label-free estimate of where a model will fail before it is deployed there. This addresses the target-validation problem that WILDS identifies as unresolved.

Core Claim

The closest existing work, Quantifying geographic domain shift to decouple the geospatial transferability of human mobility flow generation models, evaluated four flow models across 2,265 US counties with leave-one-state-out transfer. It reported two associations. Feature-distribution shift, measured by how easily an adversarial classifier separates source from target, is negatively associated with transfer performance. Change in spatial clustering, measured by multivariate Moran's I, is positively associated. That study motivates ours: it shows that shift is measurable and that it predicts failure. It also constrains us, because it found that data volume matters less than shift itself. It leaves three things unaddressed. Its design is observational, so the two shift measures covary with state size and density. It fits no interaction terms between shift components. And it never looks at what the models themselves compute.

We claim three things. First, once the design breaks the correlation between shift components, transfer loss will show significant non-additive structure. In particular, the harm from feature shift will depend on the level of spatial-clustering shift and on transfer direction. The known asymmetry of transfer (A→B good, B→A bad) is itself an interaction between the region pair and the direction. Second, a model's reliance on pairwise feature interactions, combined with how much those feature pairs' joint behavior changes in the target, will predict transfer loss beyond the existing shift measures. Third, this interaction-weighted predictor will beat a version weighted only by per-feature (additive) importance. That comparison isolates the non-linear ingredient.

Research Program

Designing the geography. A geographic experiment cannot randomize regions, but it can choose them. For every candidate source-state/target-county pair in CommutingODGen, we compute three shift components, each without using any flow labels. The first is feature-distribution shift, following the prior paper. The second is spatial-clustering shift, also following the prior paper. The third is urban morphology: whether the target is monocentric or polycentric, which that paper names as a gap in global autocorrelation metrics. We bin each component into low and high. We then select pairs to fill a balanced 2×2×2 grid, crossed with transfer direction, with an equal number of replicate pairs per cell and blocking by source state.

Balance does not remove unobserved confounders, and we will say so. It does remove the collinearity that makes interaction coefficients unestimable in the observational analysis. We borrow the analysis structure from the split-plot field trial in Statistical power analysis of a 2‐year field study and design of experiments to evaluate non‐target effects of genetically modified Bacillus thuringiensis corn. There, the hard-to-change factor (crop genotype) occupied large main plots and the easy-to-change factor (insecticide) occupied nested subplots. In our design, the geographic pair is the main plot. Model class (DeepGravity, Random Forest, gradient-boosted trees, GMEL) and training seed are subplots applied within every pair. The analysis is a mixed model with a random effect for pair. That paper also showed that single-season trials were badly underpowered, with only 22 of 56 comparisons reaching 80% power. Following that lesson, we set replicates per cell with a pilot-based power analysis for the smallest interaction effect we care about, before running the full grid.

Measuring what the model uses. For each trained source model, we compute pairwise interaction attributions over grouped input features. The 97 demographic variables and 36 point-of-interest categories are collapsed into roughly twenty semantically coherent groups plus distance. We use PolySHAP: Extending KernelSHAP with Interaction-Informed Polynomial Regression, which extends KernelSHAP (a regression-based estimator of Shapley feature credit) with explicit interaction terms. PolySHAP supplies the method here. Its Theorem 5.1 shows that paired sampling, which evaluates each feature subset together with its complement, is equivalent to a second-order polynomial fit. That equivalence gives us pairwise interactions at the cost of paired KernelSHAP. It also avoids the third-order blowup the paper flags for more than 100 features.

We define the interaction-weighted shift of a target $T$ relative to source $S$ as

$$ \Delta_{\text{int}}(S,T) = \sum_{i<j} \bar{\phi}_{ij}^{S}\, \big|\, \mathrm{MI}_T(x_i;x_j) - \mathrm{MI}_S(x_i;x_j) \,\big|, $$

Here $\bar{\phi}_{ij}^{S}$ is the mean absolute pairwise interaction attribution of feature groups $i$ and $j$ on source data, normalized to sum to one. $\mathrm{MI}_R(x_i;x_j)$ is the mutual information between groups $i$ and $j$ among tracts in region $R$. The quantity is large when the model leans on a feature pair whose co-variation changes in the target. This can happen even if each feature's marginal distribution is unchanged, a case that per-feature shift measures cannot see. The additive counterpart, $\Delta_{\text{add}}$, replaces the pairwise weights with main-effect attributions $\bar{\phi}_i^S$ and replaces the dependence change with a marginal divergence per group.

Secondary replication. The same design is repeated on WILDS-FMoW satellite land-use classification. The crossed factors are held-out region, acquisition year, and binned sensor metadata (off-nadir angle, sun elevation). The interacting groups are the image embedding and the metadata fields. Functional Map of the World reported that adding metadata to images raised average F1 from 0.679 to 0.722, so image–metadata interactions demonstrably carry signal there. This replication tests whether the claim holds beyond tabular flow models.

Baselines

The naive reference point is to predict every transfer's performance as the source model's in-distribution performance. This is equivalent to assuming no geographic effect. Any shift-based predictor must first beat it.

The strongest existing method is the linear mixed-effects model from the geographic domain shift paper. It regresses transfer CPC on feature-distribution shift and spatial-clustering shift. CPC, the common part of commuters, is the overlap between predicted and true flows. That paper reports DeepGravity as the strongest flow model, with mean CPC 0.603, versus 0.558 for Random Forest and 0.525 for GBRT. It also reports CPC above 0.70 in Midwestern states. For how well its regression predicts transfer performance out of sample, no reported result was found on Lacuna. We will therefore refit it ourselves on identical pairs and folds.

A second strong baseline augments that same regression with all pairwise interaction terms among the shift components, fit on the observational pair set. This checks whether the controlled design, rather than just a richer regression, is needed.

The key ablation replaces $\Delta_{\text{int}}$ with $\Delta_{\text{add}}$. This keeps the model-aware weighting but drops the interaction structure. We also include a matched-dimension random-weight version of $\Delta_{\text{int}}$ to rule out the possibility that any extra covariate helps.

All predictors are fit on the same pairs, evaluated with leave-source-state-out folds, and given the same hyperparameter search budget. Flow models share the same training recipe and seeds. Beating the strongest baseline means a higher held-out Spearman correlation between predicted and actual transfer CPC, with a confidence interval from pair-level bootstrap that excludes the baseline's value. It also means better recall of the bottom-quartile failure targets.

Why not just…?

Why not just add interaction terms to the existing observational regression? This is the cheapest alternative and we run it as a baseline. Across natural state pairs, feature shift and clustering shift covary with population and density, so their product term is nearly collinear with the main effects. The resulting coefficients are unstable. We will report variance inflation factors and the sign stability of the interaction coefficient across bootstrap resamples in both the observational and the balanced designs. If the observational fit yields stable interactions that agree with the factorial ones, the design adds little. We would then fall back to the interaction-weighted predictor as the sole contribution.

Why not just trust the adversarial-classifier shift measure, which already sees the full joint distribution? This is a real threat. A domain classifier can in principle detect changes in feature co-variation. Its weakness is that it weights all shift equally, whether or not the model uses the affected features. $\Delta_{\text{int}}$ counts only shift in the pairs the model relies on. The decisive test is partial: does $\Delta_{\text{int}}$ add held-out predictive value after the classifier-based shift is already in the model? If it does not, our second claim fails. We will report that outcome rather than reframe it.

Why not just vary one shift factor at a time? Varying one factor at a time holds the others at one fixed level, so it cannot see antagonism or thresholds. Agronomy shows how this misleads. In the zinc biofortification study, zinc looks ineffective if tested only alongside phosphorus. In the LLM study Manipulation Is Task-Dependent, single-axis conclusions reversed across tasks, with a cross-task rank correlation of 0.055. Our grid contains every one-factor-at-a-time contrast as a subset. We will explicitly compare the conclusions those subsets would have produced with the full-grid estimates.

Why not just fix the model with domain-invariant training instead of diagnosing it? WILDS found that robust training methods failed to beat plain empirical risk minimization on 8 of 10 real-world shifts. Extending the WILDS Benchmark for Unsupervised Adaptation found that adaptation gains were confined to two image tasks. Dual Risk Minimization and Revisiting Spurious Correlation in Domain Generalization remove context: the first via LLM-written descriptions that strip it out, the second via a frequency split. Both treat context as a separable nuisance. In commuting flows, context-by-feature interactions are plausibly the signal itself. We therefore aim to predict where failure will happen, not to erase context. One subplot arm does train with CORAL, a method that aligns source and target feature statistics. This lets us test whether invariance helps on the high-$\Delta_{\text{int}}$ cells or hurts there.

Proposed Experiments and Decisive Checks

The primary experiment fits the split-plot mixed model on the balanced CommutingODGen grid. It tests the three-way and direction interactions on transfer CPC for each model class. The first claim is supported if at least one shift-by-shift or shift-by-direction interaction is significant and stable under bootstrap. It is refuted if an additive model fits as well by held-out likelihood.

The second experiment compares transfer-risk predictors under leave-source-state-out evaluation. It follows the ordering already described: naive, then linear shift regression, then observational interaction regression, then additive-weighted shift, then interaction-weighted shift. The metrics are Spearman correlation and bottom-quartile recall.

A controlled sanity check makes the mechanism falsifiable. Within the source region, we permute one feature group across tracts within strata of its partner group's values. This destroys that pair's dependence while leaving both marginals intact. The model's in-region CPC drop should track that pair's $\bar{\phi}_{ij}$, which ties the attribution to actual failure.

The FMoW replication repeats the grid-and-predictor comparison with worst-region accuracy as the outcome. If $\Delta_{\text{int}}$ helps on commuting flows but not on satellite images, we learn the claim is modality-specific. That is still a useful, publishable boundary on it.