Measuring Partial-Overlap Leakage in Aerial Detection Benchmarks

Aerial and satellite benchmarks are built by cutting huge scenes into tiles, and the same ground is photographed again and again. A test tile can therefore share a third of its area with a training tile from another pass, at a different angle, scale or color balance, without being a duplicate in any usual sense. Current contamination tools miss this. Dataset cleaners compare whole images and look for near-copies. Contamination detectors for vision-language models change question text or regenerate the image with a diffusion model, and neither step works on oriented bounding boxes over thousands of pixels. Meanwhile, remote-sensing detectors and vision-language models report headline scores on DOTA without ever checking whether test ground was seen in training.

We propose a two-stage audit that scales to large aerial collections. A cheap whole-tile descriptor first narrows the search to a short list of candidates. Patch-level matching with a geometric consistency check then runs only on that list and estimates, for each test tile, what fraction of its area appears somewhere in the training data. That fraction sorts every benchmark image by how much of it leaked. The immediate payoff is empirical: we can measure how much of a detector's reported accuracy comes from ground it has already seen, rather than inferring leakage from how the model behaves under perturbation.

Core Claim

In tiled aerial data, leakage is mostly partial and happens at the level of regions, not whole images. Whole-image near-duplicate search, even scaled up with approximate nearest-neighbor indexes, will under-count it. Patch-level matching with geometric verification finds it, and a filter-first cascade keeps the cost close to linear in dataset size. We further predict that detection accuracy on test tiles rises with their leaked-area fraction. If so, the gap between the scores of low-leak and high-leak tiles is a direct, model-agnostic measure of contamination for detection benchmarks.

The central quantity is the leaked-area fraction of a test tile $x$:

$$ \ell(x) = \frac{\left|\bigcup_{t \in T} H_{t \to x}(t) \cap x\right|}{|x|}, $$

Here $T$ is the set of training images whose matches pass verification, $H_{t \to x}$ is the estimated geometric transform (a homography, meaning a mapping between two views of the same flat surface) that places training image $t$ onto $x$, and $|\cdot|$ denotes pixel area. The contamination gap is then the difference in detection score between tiles with $\ell(x) \geq \tau$ and tiles with $\ell(x) \approx 0$, where $\tau$ is an area threshold.

Relation to Prior Work

Two papers motivate the problem. SelfClean trains a self-supervised encoder on a dataset and flags near-duplicates, off-topic images and label errors by distances in the learned feature space. It found nearly 15% near-duplicates in Fitzpatrick17k, many of them crossing the train/test split, which shows that split leakage in public vision datasets is real and large. The same work notes that its pairwise duplicate search scales quadratically, and it treats duplicates as whole-image pairs. We keep its self-supervised, in-domain features but change both the unit of comparison (regions rather than images) and the search strategy (a cascade rather than all pairs).

RS-YOLOX evaluates detectors on sliced aerial scenes, and RSCoVLM trains a vision-language model on pooled remote-sensing corpora. Both report DOTA scores without auditing overlap between the splits they train and test on. These are the systems whose numbers we re-examine.

Two further lines of work constrain the design. Both Text and Images Leaked! detects contamination by permuting answer options and masking words in back-translated captions. Contamination Detection for VLMs Using Multi‑Modal Semantic Perturbations edits images with GPT-4o, a Flux diffusion model and ControlNet so that a different answer becomes correct, and it shows that text-only perturbation misses visual memorization. Both methods work on multiple-choice or caption tasks with small natural images, and the second relies on generation that the file notes fails on fine text and geometry. Neither applies to oriented-box detection on large scenes. Our audit is therefore data-side rather than behavioral: it needs the training data but no generative model.

Two further papers supply the methods. The classifier-first cascade of $\mathcal{R}^2$-CNN, as summarized in the file, discards about 99% of empty tiles before running an expensive detector. WSABIE (W) replaces an exhaustive k-nearest-neighbor search with retrieval in a compact learned embedding, reporting 0.17 seconds and 82 MB per image against an estimated 3,913 days and 27 GB for the exhaustive search. We reuse both ideas, but for auditing training data rather than for annotation.

Research Program

The first component is the audit itself. Stage one trains a DINO-style self-supervised encoder in the aerial domain, as SelfClean argues for, and indexes every training tile at several scales and rotations with a compact descriptor. This makes candidate retrieval sub-linear. Stage two runs only on the top-$k$ candidates. It extracts dense patch features from both tiles, matches patches mutually, fits a homography with RANSAC (random sample consensus, a method that fits a transform while ignoring bad matches), and rasterizes the verified overlap to compute $\ell(x)$. Two knobs, $k$ and the descriptor scale set, trade recall against compute, and we report the full trade-off curve rather than a single operating point.

The second component validates the audit against known ground truth. We re-tile the original large DOTA scenes with random offsets, rotations, rescaling and photometric shifts, which yields tile pairs whose true overlap is known exactly. Where the source imagery carries georeferencing, we use it as a second, independent check.

The third component measures consequences. We audit DOTA-v1.5 train against val. We also audit the pooled remote-sensing instruction data released with RSCoVLM (remote-sensing-sft-data) against the DOTA-v1.0 evaluation images and LRS-VQA-Zoom, since that corpus mixes several public benchmarks and synthetic data. We then re-score the models on test subsets grouped by $\ell(x)$.

Baselines

The naive reference point is the published benchmark score with no audit. RS-YOLOX reports 78.49% mAP50-95 on DOTA-v1.5, against 73.57% for base YOLOX (RS-YOLOX). RSCoVLM reports $AP_{nc50}$ = 55.50 on DOTA-v1.0 (RSCoVLM), where $AP_{nc}$ is average precision computed without confidence scores. These are the numbers whose inflation we estimate. We retrain YOLOX under the published recipe, since DOTA test labels are held by an evaluation server and we must work on val. The audit "beats" these reference points in the sense that matters here if it identifies a leaked subset on which scores differ significantly from the clean subset.

The strongest competing auditor is SelfClean-style whole-image near-duplicate search, scaled with an approximate nearest-neighbor index so that it runs on the same data. To keep the comparison fair, it uses the same in-domain encoder, the same GPU budget and the same threshold tuning on the synthetic re-tiling set. The file contains no reported result on Lacuna for near-duplicate recall on aerial imagery from either SelfClean or FastDup. We will therefore measure it ourselves. Beating it means higher recall of pairs whose true overlap is at least $\tau$, for $\tau$ between 0.2 and 0.8, at equal compute and equal precision. We also include brute-force keypoint matching over all pairs on a subsample that is small enough to run exhaustively. It serves as an accuracy ceiling for the cascade.

The key ablation removes geometric verification: stage two keeps patch matching but drops the homography fit. This isolates whether estimating area, rather than only retrieving candidates, is what distinguishes leaked tiles from merely similar terrain, such as two different harbors.

Why not just…?

Why not just run approximate nearest-neighbor search on global embeddings? This is the most serious threat. If most leakage in these datasets is whole-tile duplication, the approach will work and our second stage is unnecessary. The synthetic re-tiling experiment settles the question: we plot recall against true overlap fraction for both methods, then apply both to the real splits and compare how much leaked area each finds. If the two agree on the real data, we will report that and recommend the simpler tool.

Why not just split by geographic coordinates? Where coordinates exist, a spatial split is the right fix, and we use such data as ground truth. However, pooled instruction corpora crop, resize and re-encode images without keeping metadata, and several aerial sources are not georeferenced. The audit is meant for exactly those cases.

Why not just use behavioral contamination tests? Option permutation and caption slot-guessing do not apply to box regression. Diffusion-based editing is expensive per image, and the file notes it struggles with small objects and fine geometry (Contamination Detection for VLMs), which are precisely what aerial detection consists of. Behavioral tests also cannot say which regions leaked. We will nonetheless check whether our leaked subsets agree with a perturbation signal on LRS-VQA-Zoom questions, where such tests do apply.

Proposed Experiments

The first experiment tests audit quality. On the synthetic re-tiling of DOTA source scenes, we report precision and recall at each overlap threshold, and GPU-hours as the number of tiles grows by factors of ten, compared against the global-embedding and brute-force baselines.

The second experiment is a controlled injection test. We add val tiles to DOTA-v1.5 train at chosen overlap fractions and in increasing amounts, retrain YOLOX and an RS-YOLOX-style variant, and measure mAP50-95 on the injected tiles against untouched ones. This gives a dose-response curve relating overlap to score inflation, analogous to the epoch-dependent degradation reported for LLaVA (Contamination Detection for VLMs).

The third experiment applies the audit to real data. We run it on DOTA-v1.5 train/val and on the RSCoVLM instruction data against DOTA-v1.0 and LRS-VQA-Zoom, then report mAP50-95 and $AP_{nc50}$ separately for tiles with $\ell = 0$ and tiles with $\ell \geq \tau$, with bootstrap confidence intervals.

The decisive check has two parts. The injection curve must be monotone. The gap between leaked and clean subsets on the real splits must also exceed what we observe between visually similar but non-overlapping terrain, which we estimate from pairs the verifier rejects. If both hold, partial-overlap leakage is a measurable part of reported aerial detection accuracy. If the injection curve is flat, leakage of this kind is harmless for detectors, and that negative result is itself worth reporting.