Models trained on geographic data (traffic sensors, crop counties, air-quality monitors) often score well on average while failing badly in particular places. Practitioners respond by making the model local: they let parameters vary across space through mixture-of-experts routing, hypernetworks, or geographically weighted regression. Nobody has a principled rule for how local a model should be. Too local and it memorizes each region's noise. Too global and it averages away real regional differences. Generalization theory could in principle answer this, but the information-theoretic bounds that unify modern learning theory are stated for independent, identically distributed samples, and spatial data violate that assumption by construction.
This note proposes a fix that is simple to state. The "ghost sample" bound, conditional mutual information (CMI), is usually read as a statement about i.i.d. data. Its in-expectation form actually needs independence only in the random coins that decide which of two paired samples is used for training. The data inside each pair, and across pairs, can be arbitrarily dependent. We pair spatial blocks with partner blocks at a chosen distance $r$ and let a coin assign one block of each pair to training. The resulting bound then certifies the model's error at locations $r$ away from where it was trained, which is the deployment question spatial forecasters actually face. For spatially smoothed models we can bound the information term in closed form: it is how far each region's local parameters move away from what its neighbors alone would predict. That quantity is a computable, theoretically grounded score for how local a model should be. It can also be minimized directly during training.
Core Claim
Let $\tilde Z$ be a "supersample" of $m$ pairs of spatial blocks, where the two blocks of pair $i$ are separated by distance about $r$. Let $U_i \in \{0,1\}$ be independent fair coins choosing which block of each pair is trained on, and let $\Lambda_i$ be the pair of losses (clipped to $[0,1]$) that the trained model incurs on the two blocks. The evaluated-loss form of the CMI argument gives
$$\Big|\,\mathbb{E}\big[R_{\text{partner}}(r) - R_{\text{train}}\big]\,\Big| \;\le\; \frac{1}{m}\sum_{i=1}^{m}\sqrt{2\, I(\Lambda_i; U_i \mid \tilde Z)}.$$
Here $R_{\text{partner}}(r)$ is mean loss on the untrained partner blocks and $R_{\text{train}}$ is mean loss on the trained blocks. The proof conditions on $\tilde Z$ and uses only the independence of the coins. Spatial autocorrelation therefore does not invalidate the bound. Its effect is to change what the bound means, and this is made explicit through $r$.
The second claim is about computation. Suppose block $i$ has a local prediction head with a Gaussian posterior $Q_i$, as in a geographically weighted regression or a hypernetwork-generated last layer made Gaussian by the conditioning trick of Conditionally Gaussian PAC-Bayes. Then the information term is bounded by
$$I(\Lambda_i; U_i \mid \tilde Z) \;\le\; \mathbb{E}\,\mathrm{KL}\big(Q_i \,\|\, P_{-i}\big).$$
$P_{-i}$ is the head that the spatial model would assign to block $i$'s location using only data outside pair $i$, and $\mathrm{KL}$ is the Kullback–Leibler divergence. We call this the local excess information. It is near zero for a global model and large for a model that fits each region to its own data. It is the missing criterion for choosing between local and global parameterizations.
Where This Sits
The general method comes from the unified-bounds line of work. A unified framework for information-theoretic generalization bounds derives mutual-information, CMI and chaining bounds from one decorrelation lemma. Information Complexity and Generalization Bounds and Generalization Bounds via Information Density and Conditional Information Density show that PAC-Bayes and information bounds are relaxations of one change-of-measure inequality. Reasoning About Generalization via Conditional Mutual Information introduces the ghost-sample construction we reuse. All of these state their results for i.i.d. samples. The first explicitly lists non-i.i.d. data as open. We do not extend their machinery. We change what the supersample contains and which risk it certifies.
Three papers motivate the gap. Heterogeneity Matters even More in Distributed Learning extends CMI to federated clients and finds that heterogeneity shrinks the generalization gap. It treats clients as exchangeable abstract distributions with no geometry. It also concedes that population risk can still be high, which conflicts with the localized failures documented by FairSTG. Our distance-indexed partner risk resolves this tension directly. Distant, heterogeneous partner blocks raise the certified partner risk even when the conventional gap looks small, so the bound reports the regional failure that a gap bound hides. FedCR uses a CMI regularizer between local and global features over discrete clients. Our KL-to-neighbor-prior plays a similar role, but the prior is spatially predicted rather than global, and the term enters a certificate rather than serving as a heuristic.
Two papers supply the spatial setting and constrain the design. Deep Spatial Domain Generalization (SIGNN) generates local weights for unseen coordinates with a hypernetwork, without guarantees and with an unjustified isotropic Gaussian prior. Our $P_{-i}$ replaces that prior with a leave-pair-out spatial prediction. Cross-validation for Geospatial Data shows that random K-fold validation was 43% optimistic for a random forest on California housing. Its importance-weighted buffered cross-validation (IBCV) brought the estimate within 9% of the true error. This is the strongest empirical competitor. Our supersample with a minimum pair distance is the theoretical counterpart of its buffering. Finally, Benchmarking Regression Models Under Spatial Heterogeneity finds that geographically weighted regression overfits small or noisy data and that spatial random forests often fail to beat global ones. It names the missing local-versus-global criterion that we target.
Research Program
The theoretical work has three parts. The first is to prove the distance-indexed partner bound carefully, including a high-probability version over coins. We also need to verify that the per-pair comparison distribution $P_{-i}$ may legitimately depend on the other pairs' coins. That is the delicate step, and if it fails we fall back to a prior fitted on a held-out half of the pairs.
The second part derives the KL bound in closed form for kernel-weighted local linear models. There, local excess information reduces to a function of how much weight each block puts on its own data relative to neighbors, an analogue of leverage. We then extend it to hypernetwork-generated Gaussian heads.
The third part gives a worst-region variant. It bounds partner risk for each region separately, with the region's own information term, so that the certificate speaks to the performance heterogeneity FairSTG targets rather than only the average.
The practical work has two products. The first is a selection rule: choose the bandwidth, the number of experts, or the hypernetwork smoothness that minimizes training risk plus the bound. Because local excess information comes from one trained model, the rule needs no retraining. The second is a trainer. A SIGNN-style hypernetwork is trained to minimize the bound directly, by the same route Conditionally Gaussian PAC-Bayes uses for 0/1 bounds on images. The loss penalizes each region's head for straying from its neighbor-predicted head, in proportion to how little data supports the deviation.
Baselines
The naive reference point is global empirical risk minimization with the same backbone and coordinates added as inputs. On the SIGNN PM2.5 task, Deep Spatial Domain Generalization reports MAE 12.44 for standard ERM against 9.40 for SIGNN. We rerun both under our splits rather than reusing those numbers.
For localization selection, the strongest competitor is IBCV from Cross-validation for Geospatial Data, alongside plain spatial block cross-validation. It is strongest because it directly estimates risk under autocorrelation and shift. We give it the same candidate grid and the same blocks, and count training runs. Our rule uses one training run per candidate; IBCV uses K. We also report IBCV at a matched single-split budget.
For predictive accuracy, the strongest methods are SIGNN on its ten-dataset suite and Adaptive-MAML-B from Task-Adaptive Meta-Learning Framework for Advancing Spatial Generalizability. On USDA crop yield it reports overall $R^2$ 80.58% against 78.98% for MAML and 71.34% for an LSTM, and 64.39% on hard counties against 37.96%. All methods get identical hyperparameter search budgets. For worst-region error on METR-LA and PEMS-BAY, the competitor is FairSTG, which reports a 5.59% to 8.15% reduction in the spatial variance of MAE.
Two ablations isolate the key ingredients. The first replaces the neighbor-predicted prior $P_{-i}$ with a single global prior. This tests whether spatial prediction of the prior, not just shrinkage, matters. The second replaces distance-matched pairing with random pairing of individual points. This should reproduce the optimistic certificates that i.i.d. reasoning gives under autocorrelation.
Beating the baselines has a different meaning for each comparison. For selection, it means lower regret, measured as the true held-out risk of the chosen model minus the best candidate's, at fewer training runs than IBCV. For certification, it means valid bounds where i.i.d. PAC-Bayes is violated. For the bound-trained hypernetwork, it means matching SIGNN's mean MAE while improving the worst-decile region.
Why not just…?
Why not just pick the localization level with buffered cross-validation? This is the real threat. IBCV already corrects most of the optimism in random cross-validation. If it selects as well as our rule at equal compute, the selection result reduces to a cost saving. The comparison on the synthetic suite settles this, because true risk is known there. Our rule needs one fit per candidate. It also gives a certificate and a per-region explanation, local excess information, that cross-validation does not. We will report honestly if IBCV dominates.
Why not just train a global network with coordinates as inputs? The evidence in the file is against it as a general answer. Global ERM trails SIGNN on PM2.5 (12.44 versus 9.40 MAE). A global LSTM reaches 37.96% $R^2$ on hard crop counties versus 64.39% for adaptive meta-learning. Coordinate-augmented random forests did not consistently beat global ones in Benchmarking Regression Models Under Spatial Heterogeneity. The global model is the zero-information end of our grid, so our rule can choose it whenever it is in fact best.
Why not just penalize spatial error variance, as FairSTG does? A variance penalty equalizes training errors across regions. It does not tell you whether a region's low error comes from memorization, and it provides no guarantee at unseen sites. We compare worst-region partner MAE directly against FairSTG on traffic data. If FairSTG matches us there, the remaining difference is the certificate.
Why not just apply a standard i.i.d. PAC-Bayes bound to spatial models? It is likely to be falsely tight. Nearby training points make in-sample and nearby-test losses agree, which is the same mechanism behind the 43% cross-validation optimism. The random-pairing ablation measures how often such bounds are violated at distance $r$.
Proposed Experiments
The decisive first test uses the synthetic data-generating processes from Benchmarking Regression Models Under Spatial Heterogeneity. These cover $n$ from 100 to 5000, varying non-stationarity and noise. Because true risk at distance $r$ can be computed, we can check three things. First, whether the bound holds at every $r$. Second, whether local excess information rises exactly where geographically weighted regression overfits, at small $n$ and high noise. Third, whether the selection rule has lower regret than IBCV and block cross-validation.
On the five real datasets in that suite and the California housing data used by IBCV, we test the agreement between the certified partner risk and buffered held-out error. On the SIGNN suite (PM2.5, temperature, influenza, civil unrest) and on USDA crop yield, we test whether the bound-trained hypernetwork matches mean accuracy while improving worst-decile regions. Traffic benchmarks serve only as a secondary check of the per-region certificate.
The proposal fails in two cases. If the KL surrogate is vacuous at realistic block counts, the certificate has no practical value. If selection by the bound is no better than a single buffered split, the selection rule adds nothing.
References
- Conditionally Gaussian PAC-Bayes
- A unified framework for information-theoretic generalization bounds
- Information Complexity and Generalization Bounds
- Generalization Bounds via Information Density and Conditional Information Density
- Reasoning About Generalization via Conditional Mutual Information
- Heterogeneity Matters even More in Distributed Learning: Study from Generalization Perspective
- FairSTG: Countering Performance Heterogeneity via Collaborative Sample-Level Optimization
- FedCR: Personalized Federated Learning Based on Across-Client Common Representation with Conditional Mutual Information Regularization
- Deep Spatial Domain Generalization
- Cross-validation for Geospatial Data: Estimating Generalization Performance in Geostatistical Problems
- Benchmarking Regression Models Under Spatial Heterogeneity
- Task-Adaptive Meta-Learning Framework for Advancing Spatial Generalizability