Regional Radioactive Marks: Measuring Which Regions a Geographic Model Relies On

Models trained on geographic data often fail when moved to a new region, but there is no cheap way to see, for a given test location, which training regions the model is actually relying on. We propose to borrow the logic of radioactive tracer mapping. In a physical tracer study, a substance is labelled with a radioactive isotope and its accumulation is imaged. In our version, each training region's images are labelled with a faint, region-specific signature hidden in the model's feature space. After training, we read out how strongly each signature appears in the parameters the model uses at a given location. That readout gives a per-location map of regional reliance. It costs one training run instead of one retraining per held-out region.

The digital half already exists. Radioactive data: tracing through training shifts training images slightly along a secret "carrier" direction in a network's feature space, at about 42 dB peak signal-to-noise ratio, so the images look unchanged. Models trained on the marked data then carry a detectable trace: p < 10⁻⁴ with only 1% of the data marked, at an accuracy cost of 0.1–1.0%. That work asks a yes/no question about one dataset on ImageNet-style classifiers. We ask a graded, spatially resolved question, using many carriers at once in models whose parameters change with location. Our claim is that the alignment between location-specific parameters and region-specific carriers predicts how much a location's accuracy would fall if a region were removed from training. If so, it is a usable instrument for diagnosing geographic scope limitation.

Core Claim

Suppose a model's prediction at location $s$ uses a classification head $W_s$, with one row $w_{s,c}$ per class $c$. Suppose region $r$'s training images of class $c$ were marked along a carrier $u_{r,c}$, a random unit vector in the $d$-dimensional feature space. We define the reliance of location $s$ on region $r$ as

$$ \rho_r(s) = \frac{1}{C}\sum_{c=1}^{C} \frac{w_{s,c}^\top u_{r,c}}{\lVert w_{s,c}\rVert}, $$

where $C$ is the number of classes. Under the null hypothesis that region $r$ left no trace, each term is approximately distributed like the cosine between a fixed vector and a random direction in $d$ dimensions. This gives a closed-form p-value, the same test used in Radioactive data: tracing through training.

Carriers must be specific to each class within each region, not just to each region. A direction shared by all classes in a region does not help classification, so the network has no reason to keep it. The radioactive-data paper makes exactly this argument against ordinary watermarks. The falsifiable prediction is that the ordering of regions by $\rho_r(s)$ matches the ordering of regions by actual leave-one-region-out damage at $s$. It should also do better than cheap proxies such as geographic distance or feature similarity.

Why This Intersection, and What the Prior Work Contributes

Geographic scope limitation is well documented, but the evidence is aggregate. Deep Spatial Domain Generalization introduces SIGNN, which uses a hypernetwork to generate location-specific parameters $W_s$ from directional neighbor graphs. It cuts Los Angeles PM2.5 error from 12.44 MAE under pooled training to 9.40. CoDEx combines regional expert heads through a learned selection network and reaches 53.9% on Functional Map of the World against a 51.6% baseline. Positional Encoder GNNs add multi-scale coordinate encodings and an auxiliary loss for local spatial autocorrelation. All three improve held-out-region accuracy, and all three motivate this proposal. None of them tells a user which source regions a particular prediction draws on, or warns when a location is being served by an inappropriate region. Their location-dependent parameters are exactly what makes a per-location readout possible, so they also supply the object we measure.

The radioactive-data paper supplies the method and also constrains it. It reports that black-box detection needs 10–20% of the data marked, and that an informed trainer can strip the carrier by analyzing the feature subspace. We therefore restrict this work to white-box readout by the model owner, as an auditing tool rather than an ownership test.

The physical tracer literature supplies the experimental controls we copy, not just the metaphor:

  • Specificity versus off-target signal. Head-to-head autoradiography of three tau tracers in Agüero et al. separated specific from off-target binding. Neuromelanin binding could be displaced by harmine but not by monoamine oxidase inhibitors. This "displacement" design is the model for one of our controls.
  • Signal that reflects the tissue, not the target. Oliveira et al. show that PSMA tracers light up reactive astrocytes around gliomas. That uptake is real signal but does not indicate the target. The analogous danger for us is a carrier signal that tracks visual similarity rather than reliance.
  • Freezing the map before reading it. Cryogenic autoradiography in Mori et al. halts hydrogen diffusion so the map reflects where atoms were trapped, not where they drifted. The analogue is that fine-tuning after training may "wash" weak traces. We test this directly.

Research Program

Benchmarks and marking. We use Functional Map of the World and DynamicEarthNet, the two benchmarks on which CoDEx reports numbers. Both are satellite imagery with geographic metadata, so image-space marking applies directly. Functional Map of the World is partitioned into its standard continental regions, and we also cut finer grid cells to stress the method with more carriers. Marks are made against a fixed pretrained backbone, following the radioactive-data recipe. We sweep the fraction of each region marked from 1% to 100% and keep the image fidelity budget near 42 dB. With $K$ regions and $C$ classes we need $KC$ nearly orthogonal carriers. This is easy at the backbone's feature width for continental regions, and it becomes a measured limit for fine grid cells.

Models read out. We read out three kinds of model. The first is a pooled classifier with a single global head, where $W_s$ does not vary with location. The second is CoDEx, where $W_s$ is the selection-weighted combination of expert heads for each test image. The third is a SIGNN-style hypernetwork head attached to the image backbone, so that $W_s$ is generated from coordinates. SIGNN was originally evaluated on tabular spatial data rather than images, so this third model is our construction, and we report its accuracy alongside the original.

Ground truth. For each region $r$ we retrain each model without $r$. We record the change in loss at every held-out test location. This "leave-one-region-out damage" is the quantity the readout must predict. It is expensive, but it is affordable at continental granularity, and it is precisely the cost our method aims to replace.

Baselines

Naive reference points. Geographic distance from test location $s$ to the centroid of region $r$ is the zero-cost answer to "which region does $s$ rely on." The pooled model's readout is the second naive reference. Its single head cannot vary by location, so its per-location ranking is constant up to class composition, and any location-resolved skill must come from location-dependent parameters. Beating distance means a higher per-location Spearman correlation with leave-one-region-out damage, averaged over test locations, with a paired bootstrap interval over locations that excludes zero.

Feature similarity. For each region we compute the mean similarity between the test image's backbone features and that region's training features, or equivalently the output of a classifier trained to predict region from features. We consider this the strongest cheap competitor, because similarity often co-varies with reliance. It runs on the same backbone and the same models, with no extra tuning. This is the comparison that determines whether the proposal is worth it.

Expert-selection weights. For CoDEx we also read the selection network's expert weights directly. This is fair only where experts map onto regions, and only within that one architecture.

Gradient-based influence. We sum training-gradient alignments over each region's examples at saved checkpoints. This is the most principled and the most expensive competitor. We give it the same compute budget as one additional training run and report its cost.

Ablation of the key ingredient. We compare class-specific carriers $u_{r,c}$ against a single carrier per region. The expected result is that a single carrier is discarded by the network and loses signal. A second ablation randomizes carrier assignment after training, which gives the empirical null.

No reported result found on Lacuna for the correlation between any readout and leave-one-region-out damage. The strongest reported task numbers we must not degrade are CoDEx at 53.9% average accuracy on Functional Map of the World and 39.1 mIoU on DynamicEarthNet. For SIGNN, the reported number is 9.40 MAE on Los Angeles PM2.5.

Why not just…?

Why not just use feature-space similarity? It is cheap and may correlate strongly with reliance. That makes it a real threat, not a strawman. The two quantities come apart wherever similar-looking regions contribute unequally. A region can look similar but be redundant because another region covers the same appearance, or look dissimilar but supply the only labels for a rare class. Our leave-one-region-out ground truth measures reliance directly, so the comparison settles the question. If similarity matches the carrier readout within the bootstrap interval, we will report that it does, and we will recommend similarity as the cheaper instrument.

Why not just retrain without each region? That is the ground truth, and it costs $K$ retrainings per model. That is feasible for seven continents and infeasible for hundreds of grid cells or a frequently updated production model. The proposal only matters if one marked training run recovers most of the ranking information. We quantify this by the readout's correlation with damage as the number of regions grows.

Why not just read CoDEx's expert-selection weights? They exist only in one architecture. They describe which head is used, not which data shaped that head, and every head shares the backbone trained on all regions. We report this baseline wherever it applies. If it matches our readout on CoDEx, the method's value narrows to architectures like the pooled and hypernetwork models.

Why not just use gradient-based influence? It needs stored checkpoints and per-example gradients. Its cost grows with dataset size rather than with the number of regions. We run it under a matched budget. If it wins at matched cost, the carrier method is dominated.

Proposed Experiments and Decisive Checks

Main test. We report per-location Spearman correlation and top-1 agreement between $\rho_r(s)$ and leave-one-region-out damage, on both benchmarks and all three model types, against all baselines above. We also report p-value calibration under the randomized-carrier null and the accuracy cost of marking. That cost should stay within the 0.1–1.0% range reported for radioactive data. If it does not, the instrument fails at the outset.

Displacement control, modelled on the harmine experiment. We fine-tune on unmarked copies of region $r$. If $\rho_r$ measures reliance, the readout should fall only to the extent that the new copies overwrite the old ones, and the damage-at-$s$ ranking should move with it. If $\rho_r$ is driven by feature similarity, the readout should persist unchanged. This control directly tests the off-target worry.

Washing test. We fine-tune briefly on a new region and track how fast each carrier decays. This mirrors the diffusion that cryogenic exposure prevents. The result tells practitioners when a readout is stale.

Scope-limitation use case. We test whether low maximum reliance across all regions flags test locations with high error. The comparison is against distance-based flags, measured by area under the error-detection curve, on the held-out continent of Functional Map of the World.

Kill condition. If class-specific carriers fail to beat feature similarity on correlation with leave-one-region-out damage in either benchmark, we conclude that the tracer adds nothing over similarity for geographic auditing. We will report that as the finding.