Environmental health studies test hundreds or thousands of neighborhood exposures, such as temperature, green space, deprivation and air pollution, against a health outcome. To keep chance findings out of the literature, they borrow a safeguard from genetics. They screen candidates in one group of patients, the discovery cohort, and then re-test only the survivors in a second, independent group, the replication cohort. That safeguard assumes the two cohorts really are independent. With geographic data they usually are not. A patient in the replication cohort who lives next door to a patient in the discovery cohort shares the same unmeasured local conditions, so a spurious association can "replicate" simply because it was measured twice on the same neighborhood. The obvious fix is to split by region, but it overcorrects. North and south differ in so many unmeasured ways that real effects fail to replicate.
We propose to choose the split by measuring leakage directly rather than by guessing. Before any exposure is screened, we fit the outcome on design covariates only and ask how strongly residuals remain correlated between discovery and replication patients who are close under some notion of distance. We compare several candidate distances: plain geographic distance, and "spatial-attribute" distances that also count two places as far apart when their background characteristics differ sharply, as happens across a river, a ridge or a land-use boundary. The distance that best predicts residual correlation defines the split. We then partition the cohort so that cross-cohort residual correlation is near zero, while keeping the two cohorts' exposure distributions overlapping enough for true effects to replicate. The contribution is a design rule and a pre-screening diagnostic for multi-stage association studies. On semi-synthetic benchmarks with known ground truth, we will test whether it lowers false replications without the power loss of coarse regional blocking.
Core Claim
We make three claims. First, the right split for a spatial discovery–replication study is one that makes the two cohorts conditionally independent in their outcome residuals, not one that separates them by an arbitrary geographic rule. Second, this independence can be checked, and the split tuned, before screening begins, using only the outcome and non-screened design covariates, so the multiple-testing guarantee is not compromised. Third, which distance to use, whether geographic, augmented with attributes, or path-based, is an empirical question the same diagnostic can answer. The proposal therefore does not bet on a particular fused metric. It supplies the criterion that decides between them.
The motivating evidence comes from two places. SpaCE: The Spatial Confounding Environment, a benchmark of 124 semi-synthetic spatial confounding datasets, shows that random splits of spatial data leak badly enough that standard machine learning looks near-perfect on validation and then fails on held-out geography. It replaces random splits with breadth-first-search partitions of a geographic adjacency graph. That is the closest existing work. It differs from ours in two ways: it separates cohorts purely by adjacency, and it targets generalization of causal estimators rather than the false-replication rate of a two-stage screen. The second source is the two-phase exposome-wide design of External Exposome Factors and Adverse Heart Failure Outcomes in the OneFlorida+ Network, which screened 1,308 county-linked exposures with a least absolute shrinkage and selection operator (LASSO) regression in discovery and confirmed hits with mixed-effects models in replication. It does not say how spatial structure shaped its split. That is the application regime we want to make trustworthy.
Research Program
The method has three components. The first is a family of candidate distances between patients $i$ and $j$. Geographic distance is the reference. The augmentative distance appends standardized background attributes to coordinates. The multiplicative distance stretches geographic distance by attribute dissimilarity. Both variants come from Bilateral blue noise sampling, which used them to keep sample points on opposite sides of a sharp color or lighting edge from competing with each other. We borrow that construction to keep patients on opposite sides of an environmental boundary from counting as neighbors. The last candidate is the Intrinsic Spatial Distance of Evaluation of a Spatial Relationship by the Concept of Intrinsic Spatial Distance, which is a shortest path over a neighborhood graph that penalizes abrupt attribute changes along the way. That paper showed the path-based distance links elevation-matched climate stations that plain distance separates. We also include a learned variant in the spirit of the spatial-attribute proximity network of Spatial and Attribute Neural Network Weighted Regression, which maps distance and attribute differences through dense layers to a single proximity. That paper used a single attribute, wind direction. We extend it to several background attributes, fit with a sparsity penalty so that irrelevant attributes drop out. The background attributes are fixed in advance and exclude every exposure that will be screened, so building the distance cannot peek at the hypotheses under test.
The second component is the leakage diagnostic. Let $r_i$ be the residual of the outcome after regressing it on design covariates, such as age, sex and year, for patient $i$. For a split $z$ into discovery set $D$ and replication set $R$, and a distance $d$, define the cross-cohort residual correlation at range $h$:
$$C_z(h;d) = \frac{1}{|P_h|}\sum_{(i,j)\in P_h} \frac{r_i r_j}{\hat\sigma^2}, \qquad P_h=\{(i,j): i\in D,\ j\in R,\ d(i,j)<h\},$$
where $\hat\sigma^2$ is the residual variance. The distance is chosen as the one whose within-cohort residual correlogram decays most cleanly with $d$, meaning it has the best fit of a monotone covariance model under cross-validation. The split is then chosen to drive $C_z(h;d)$ toward zero at every range where the correlogram is non-negligible.
The third component is the partitioner. It works on the chosen distance's neighborhood graph and uses a balanced graph cut plus an exclusion buffer. Patients within a buffer distance of the cut are dropped from both cohorts. The objective adds a penalty on distributional mismatch between $D$ and $R$ in the screened exposures, so that each exposure's range is represented in both cohorts. The result is a single tunable trade-off between leakage and overlap, and we report the whole trade-off curve rather than one point on it.
We will analyze when the approach should help. If the unmeasured confounder is smooth in coordinates, all distances coincide and the geographic split is sufficient. The attribute-aware distances should matter only when the confounder has sharp boundaries that background attributes track. The benchmark experiments manipulate exactly this condition.
Baselines
The naive reference point is a random split of patients into discovery and replication cohorts, which is the default in two-stage pipelines. It shows how much spurious replication spatial autocorrelation produces when nothing is done.
The strongest existing split is SpaCE's breadth-first-search geographic partition, the only spatially aware splitting procedure in the file. We run it on identical data with the same discovery and replication sizes, the same screening pipeline (LASSO discovery followed by mixed-effects replication, as in the OneFlorida+ design) and the same significance thresholds. For its false-replication rate under a two-stage screen we found no reported result on Lacuna, because SpaCE reports estimator performance, not replication rates. We therefore rerun it ourselves rather than quote it. Beating it means a lower false-replication rate among null exposures at equal or higher replication power for true exposures, or equal false replication with higher power. A gain that appears only on some regions of the trade-off curve counts as partial.
A model-based baseline keeps the random split and removes spatial confounding inside the regression instead. It uses the spatial+ residualization of A simplified spatial+ approach to mitigate spatial confounding in multivariate spatial areal models, which regresses covariates on spatial basis functions before the outcome model. In its simulation that paper reduced a confounded estimate from −0.3207 to −0.1432 against a true −0.1500 and restored interval coverage from 0% to 90–100%. SpaCE likewise reports spatial+ as the top performer for binary treatments. Every split method is also run with spatial+ added, so that we measure whether splitting adds anything beyond modeling.
Two ablations isolate the key ingredient. The first uses our partitioner and buffer with plain geographic distance, which separates the effect of the diagnostic-driven cut from the effect of attribute-aware distance. The second clusters coordinates concatenated with attributes by k-means and assigns whole clusters to cohorts, which is the cheapest possible spatial-attribute split. All methods receive the same tuning budget, a fixed grid of buffer widths and balance weights, selected only by the pre-screening diagnostic and never by replication outcomes.
Why not just…?
Why not just split by geographic blocks, as SpaCE does? In many settings this may be enough, and our design is built to show when. If unmeasured confounding is smooth in space, geographic blocks already achieve near-zero cross-cohort residual correlation, and our diagnostic will select geographic distance. The approach then reduces to a principled choice of buffer width. The claim that attribute-aware distances help rests on boundary-type confounding, the pattern Leung et al. document with elevation-matched stations. Blocks by region also risk losing true effects when the two regions barely overlap in exposure, which the balance penalty targets directly. The benchmark conditions with smooth confounding versus boundary confounding settle both questions.
Why not just keep the random split and adjust for space in the model, with spatial random effects or spatial+? Model-based adjustment is a real threat and may dominate when the confounder has the structure its basis assumes. Two considerations keep splitting worthwhile. Spatial+ removes only confounding that is a function of coordinate-derived bases, and its authors flag non-linear settings as a limitation. Plain spatial random effects can also be worse than nothing: the spatial+ simulation shows a standard intrinsic conditional autoregressive model giving over 100% bias with zero coverage. Beyond this, a split guards the replication step even when the screening model is misspecified, and it works with any discovery method, including LASSO. We test adjustment alone, splitting alone and the two combined. If spatial+ with a random split matches our split, we will report that as the recommended practice.
Why not just cluster on coordinates plus attributes with k-means and assign clusters to cohorts? This is the augmentative bilateral distance in its crudest form, and it may capture most of the benefit. We did not want the proposal to depend on the path-based distance beating it, which is why the core contribution is the diagnostic rather than any particular distance. The k-means ablation is run at matched cohort sizes. If it ties the Intrinsic Spatial Distance, the conclusion is that cheap attribute augmentation suffices, and the diagnostic is still what tells an analyst whether any attribute augmentation is needed at all.
Proposed Experiments
The main testbed is SpaCE, whose semi-synthetic outcomes carry known effects and known, spatially structured, unobserved confounding. To each dataset we add screening panels of several hundred candidate exposures. The panels include a few with true effects, many nulls drawn from real spatially autocorrelated covariates in the same dataset, and nulls built to correlate with the hidden confounder, some smoothly and some with sharp boundaries aligned to observed land-use or socioeconomic attributes. For every split method, with and without spatial+, we run the full two-stage pipeline and record four quantities: the false-replication rate among nulls under Bonferroni-corrected replication, replication power for true exposures, bias of replicated effect sizes, and the pre-screening cross-cohort residual correlation. The decisive check is whether low measured leakage before screening predicts low false replication after screening across datasets. That test validates the diagnostic independently of which distance wins. We also report runtime, since shortest-path and pairwise neural distances scale quadratically without neighbor truncation, a limitation both source papers flag.
For real data, we use the England Middle Layer Super Output Area dataset from A Spatial Case-Based Reasoning Method for Healthy City Assessment. It covers 1,039 small areas across 29 cities with built-environment, walkability and air-pollution measures, from which we build an ecological exposome screen against life expectancy. Because the truth is unknown there, the checks are negative controls, namely exposures shuffled within cities that should never replicate, plus stability of the replicated set across repeated splits. Where access allows, the same protocol applies to the OneFlorida+ heart-failure data. There the reported maximum-May-temperature association, adjusted odds ratio 1.04 (95% CI 1.02–1.06), serves as a finding whose survival under leakage-calibrated splitting we test. The proposal succeeds if, on SpaCE, the diagnostic-chosen split reaches the false-replication rate of geographic blocking with measurably higher power under boundary confounding, and matches geographic splitting under smooth confounding. It fails informatively if model-based adjustment with random splits matches it everywhere.
References
- SpaCE: The Spatial Confounding Environment
- External Exposome Factors and Adverse Heart Failure Outcomes in the OneFlorida+ Network: Retrospective Cohort Study
- Bilateral blue noise sampling
- Evaluation of a Spatial Relationship by the Concept of Intrinsic Spatial Distance
- Spatial and Attribute Neural Network Weighted Regression for the Accurate Estimation of Spatial Non-Stationarity
- A simplified spatial+ approach to mitigate spatial confounding in multivariate spatial areal models
- A Spatial Case-Based Reasoning Method for Healthy City Assessment: A Case Study of Middle Layer Super Output Areas (MSOAs) in Birmingham, England