Bird-sound foundation models are trained on over a million citizen-science recordings from archives like Xeno-Canto and iNaturalist. They are then scored on "soundscapes": long passive recordings from fixed microphones at field sites. Two recent results pull against each other. Making the network bigger barely helps when models move from clean, targeted recordings to noisy soundscapes. Yet teaching the network to also predict where and when a recording was made gives large gains on a global soundscape benchmark. This note proposes a simple test of what that second gain is made of. We would retrain models after deleting every training recording within a chosen radius of the evaluation sites, sweep that radius, and trace how benchmark scores and scaling curves change. If the metadata gains and the remaining scale gains shrink sharply as the nearest training data moves away, then part of what the field calls representation learning is geographic proximity leakage. That is the spatial analogue of test questions leaking into a language model's pre-training text.
The opportunity is that contamination research has been almost entirely about discrete text, while ecological models are used exactly where they have never heard a local recording: tropical sites with little data, new deployments, new years. An honest estimate of accuracy as a function of distance to the nearest training data is what practitioners need, and no benchmark currently reports it.
Core Claim
We claim that the gain from metadata-predicting auxiliary heads, and the flat shape of bioacoustic scaling curves, both depend on how close the evaluation site is to the training data. Concretely, we predict three things. The gain from location and season heads will be concentrated at sites with same-species training recordings nearby. Once a geographic exclusion buffer removes those neighbours, the gain will shrink much more than an equal-volume random deletion would cause. And adding more distant training data will not "wash out" the proximity advantage. That last prediction runs against what How much can we forget about Data Contamination? found for leaked text, where extra novel training data erased the benefit of repeated benchmark questions. Our reasoning is that nearby recordings do not leak test items. They leak local acoustic context: regional song dialects, background noise at the site, and the co-occurring species.
The measured quantity is a proximity gap. We define it as $G(r) = A(0) - A(r)$, where $A(r)$ is the evaluation score (ROC-AUC, the area under the curve of true-positive rate against false-positive rate) of a model retrained with all training recordings within $r$ kilometres of any evaluation site removed. We always report it against a volume-matched control that removes the same number of recordings chosen at random from far away.
Relation to Prior Work
The motivating tension comes from two papers. BIRB trains retrieval models on Xeno-Canto and tests on soundscapes from New York, the Sierra Nevada, Hawai'i, Colombia, Costa Rica and Peru. It finds that growing a Conformer from small to large does not close the gap between targeted recordings and soundscapes. MetaPerch adds heads that predict location (coarse S2 cells, a hierarchical grid of roughly 50,000 km² cells at level 7), season and background species. It reports a jump on WABAD from 0.741 to 0.811 linear-probe ROC-AUC. MetaPerch never checks how close its training recordings are to the evaluation sites. We do not propose a new metadata method. We measure whether that jump survives geographic separation.
Perch 2.0 supplies the training recipe we reuse: an EfficientNet-B3 trained with supervised labels on about 1.54M recordings. It likewise does no proximity audit.
The methodology comes from contamination work, which we move from text to continuous space. How much can we forget about Data Contamination? inserts benchmark text at controlled frequencies and measures forgetting as more novel data is added. We run the reverse intervention, deletion by distance, and ask the same forgetting question. Revisiting Scaling Laws for Language Models argues that redundancy in training data bends scaling curves. Citizen-science archives cluster densely around cities, which is exactly that kind of redundancy.
Two papers show that spatial and temporal evaluation leakage is real outside text. A Critical Audit of Spatiotemporal Forecasting Benchmark Datasets and Baselines shows that forecasting benchmarks are dominated by autocorrelation, so that tuned SARIMA baselines beat graph networks. It does not study classification, scaling or distance buffers. CrypticBio builds a species holdout, COMMONUNSEEN, to guard against models having already seen the evaluation data. It does not check whether other training images came from the same coordinates.
One constraining result must be faced directly. Presence-Only Geographical Priors shows that a location prior helps legitimately, raising iNaturalist 2018 accuracy from 50.17% to 58.44%. Knowing a species' range is real knowledge, not leakage. Our design therefore separates range knowledge, which survives a buffer of a few hundred kilometres because ranges are larger than that, from site-level context, which does not.
Research Program
The first stage is an audit using existing and freshly trained checkpoints. For every evaluation clip in BirdSet, WABAD and the BIRB regions, we record four things about the nearest training recording of the same species: its great-circle distance from the clip, the time gap between the two recordings, whether both share a recordist, and whether they fall in the same S2 cell. We then stratify per-species scores by these variables. This stage is cheap, and its outcome decides whether retraining is warranted.
The second stage is the causal intervention. We train an EfficientNet-B3 on the Perch 2.0 and MetaPerch training corpus with exclusion buffers of 0, 10, 50, 200 and 1000 km around all evaluation sites. Each buffer is run with and without the metadata heads, and each has a volume-matched random-deletion control. A third arm keeps the metadata heads but shuffles the location and season targets across recordings. This preserves the multi-task regularization while destroying the geographic information.
The third stage asks whether scale interacts with proximity. We train EfficientNet-B0, B1 and B3 at 10%, 30% and 100% of the data, but only at buffers of 0 and 200 km. We then fit separate scaling curves for near and far conditions. A final "dilution" arm holds the near-site recordings fixed and adds increasing amounts of far-away data. This tests directly whether spatial leakage is forgotten the way leaked text is.
Compute is dominated by the roughly fifteen B3 runs at full scale. Every arm uses the same number of optimizer steps and the same hyperparameters, tuned once on the unbuffered run.
Baselines
The naive reference point is the published protocol: an unbuffered model scored on the standard splits. Its reported numbers are what the field currently treats as progress. For WABAD this is MetaPerch's 0.811 linear-probe ROC-AUC against 0.741 for its baseline. For BirdSet it is MetaPerch's 0.906 mean ROC-AUC and Perch 2.0's 0.908. For BEANS it is MetaPerch's 0.870 classification accuracy and 0.512 detection cmAP, and Perch 2.0's 0.504 mAP. We retrain the unbuffered B3 ourselves under our fixed budget and check that it reproduces these numbers within noise before comparing anything.
The strongest competing method is MetaPerch itself: metadata heads trained on the full corpus. It is the strongest competitor because it holds the best reported WABAD and BEANS results, and it is the method whose gain our hypothesis says is partly proximity. We run it with identical data at every buffer.
We "beat" MetaPerch only in the sense of explaining it. The proposal succeeds if the metadata-minus-no-metadata gain on WABAD at a 200 km buffer falls below half of its unbuffered value, and the volume-matched random-deletion control shows no comparable drop. If the gain is unchanged at 200 km, MetaPerch's claim of representation learning stands, and we report that.
The key ablation is the shuffled-target head. If shuffled location labels recover most of the gain, the benefit is generic multi-task regularization rather than geography. Neither proximity leakage nor a useful geographic signal would then be the explanation.
As a secondary check, a decoupled test-time geographic prior in the style of Presence-Only Geographical Priors tells us whether range knowledge alone, added after the fact, accounts for the far-site gains.
Why not just…?
Why not just use BIRB's held-out regions? BIRB already reports cROC-AUC of about 0.65 to 0.80 on species never seen in training. A held-out region, however, is a single yes-or-no split. It mixes distance with everything else that makes a region hard, such as tropical species richness, sparse labels and different recording hardware. The buffer sweep holds the evaluation sites fixed and varies only proximity, which yields a dose-response curve rather than one confounded contrast.
Why not just stratify existing checkpoints by distance and skip retraining? This is a real threat to the cost-benefit case, and it is our first stage. The weakness is that sites near training data are also temperate, heavily sampled and acoustically easier, so a correlation with distance cannot be read as leakage. If stratification shows no distance trend at all, we stop and report a negative result. If it shows a trend, only deletion with a volume-matched control can separate proximity from site difficulty.
Why not just deduplicate near-identical audio, as text contamination work does? Spatial leakage is mostly not duplication. Nearby recordings of the same population share dialect and background noise, yet they are different waveforms and would survive a similarity filter. We will run acoustic-embedding deduplication as an extra arm. We expect it to remove far less of the proximity gap than a 50 km buffer, and if it matches the buffer, the cheaper fix wins.
Proposed Experiments
The decisive plot shows WABAD and BirdSet ROC-AUC against buffer radius, with four curves: metadata heads, no heads, shuffled heads and random-deletion control. Leakage predicts a steep drop between 0 and 50 km for the metadata curve alone, then a plateau once range knowledge is the only thing left.
A second check compares scaling exponents fitted separately at 0 and 200 km across the B0 to B3 and data-fraction grid. If the far-condition curve is steeper, flat bioacoustic scaling is partly an artifact of evaluating where training data is already dense. If both curves are equally flat, the diminishing-returns wall BIRB reports is genuine.
The dilution arm settles the forgetting question. We add far-away data in steps of up to about three times the corpus size by oversampling under-represented regions, and measure how the near-site advantage changes. If that advantage persists while leaked text is forgotten under comparable dilution, geographic leakage is a more stubborn form of contamination than the text literature assumes.
BEANS, which contains many non-avian and marine tasks unlikely to overlap geographically with Xeno-Canto, serves as a negative control. Scores there should not move with buffer radius.
References
- How Much Can We Forget about Data Contamination?
- BIRB: A Generalization Benchmark for Information Retrieval in Bioacoustics
- MetaPerch: Learning from metadata for bioacoustics foundation models
- Perch 2.0: The Bittern Lesson for Bioacoustics
- Revisiting Scaling Laws for Language Models: The Role of Data Quality and Training Strategies
- A Critical Audit of Spatiotemporal Forecasting Benchmark Datasets and Baselines
- CrypticBio: A Large Multimodal Dataset for Visually Confusing Species
- Presence-Only Geographical Priors for Fine-Grained Image Classification.