Where the Pixels Go: A Compute-Matched Test of Foveated Sampling for Tiny Object Detection

Detecting objects that are only 8 to 16 pixels across in large aerial images forces a choice about where to spend resolution. A detector can shrink the whole frame and lose the objects. It can cut the frame into full-resolution tiles and pay for every patch of empty field. Or it can look closely only where objects are likely to be, which is the idea behind foveation, cluster-based cropping, and saccade-style search. Foveation here means sampling densely at a chosen point and coarsely around it, as the eye does. The adaptive options report large wins: roughly 40% fewer forward passes than grid tiling, and an 8× speedup on gigapixel scenes. Each of those results, however, comes with its own backbone, its own crop and overlap rules, its own box-merging step, and its own definition of cost. So nobody currently knows whether where you sample matters, independent of what detector you run on the samples.

This note proposes a benchmark protocol that fixes the recognizer and varies only the sampling plan. A sampling plan is a set of fixed-size glimpses, either rectangular crops or warped foveal views, chosen under a pixel budget. The protocol also includes an oracle plan that knows where the ground-truth objects are. That oracle converts the question into a measurable quantity: the share of the tiny-object accuracy gap that allocation could close in principle, and the share that current adaptive methods actually close. We also contribute one space-variant plan designed around the known blind spot of density-driven search, which is isolated objects. The plan is tested against the baseline most likely to embarrass it: uniform tiling with a cheap filter that skips empty tiles.

Core Claim

Published comparisons between tiling, clustering, and foveated detectors on small objects confound two things. One is allocation: which pixels reach the recognizer, and at what magnification. The other is recognition: the backbone, the loss, how training examples are assigned to boxes, and the box-merging step. Our claim is that once recognition and compute are held fixed, the value of space-variant sampling on tiny objects can be read off a single curve, detection accuracy against pixel budget, bounded above by an oracle plan. We also expect that most of the reported gains of adaptive methods over tiling will shrink to what a simple skip-empty-tiles policy already achieves, with one exception: isolated targets in large empty backgrounds. On those, bottom-up saliency-triggered foveation should hold a real advantage. If the oracle gap turns out to be small, the field's architectural effort belongs in the recognizer, not the sampler. That result would be worth publishing as well.

Why this is not already done

The closest works each hold one direction fixed. Clustered Object Detection in Aerial Images (ClusDet) proposes cluster regions, normalizes their scale, and detects on the resulting chips. It compares against Faster R-CNN and grid tiling under its own training setup and says that isolated objects remain an open problem. SaccadeDet regresses multi-scale density on a downscaled image and then detects on high-density patches. It is evaluated on PANDA and Camelyon, not on the standard tiny-object aerial benchmarks, and it permanently discards anything that fails to produce a density peak. These two papers motivate our allocation-versus-recognition split, because both bundle a new sampler with a new detector.

On the foveation side, FOVI does run pixel-matched and compute-matched comparisons. It finds an inverted-U curve: moderate foveation beats uniform sampling at equal pixel budget. But FOVI covers only ImageNet classification with a fixed central gaze. SpiralFovea chooses where to place tokens by computing local image entropy (a measure of how much detail a region contains) at cost linear in image size, reports token and FLOP savings on fine-grained classification, and notes that supervised-pretrained ViTs degrade on non-grid inputs. FLIP (Looking Locally) samples foveal patches around a user prompt and holds up on objects down to 0.0001% of image area on the synthetic ObjaScale benchmark. It is prompted segmentation, though, not unprompted multi-object detection. These three papers supply our glimpse mechanisms and our pretraining concern. We borrow their compute-matching discipline but apply it to dense detection, where no fixation prompt is given.

On the benchmarking side, Revisiting Point Cloud Shape Classification showed that unifying training protocols lifted PointNet++ from 90.0% to 93.3% and let a 0.8M-parameter baseline beat newer architectures. That paper supplies the method: freeze every confound except the one under test. Detecting tiny objects in aerial images supplies the clean data (AI-TOD-v2, with over 50,000 missing labels added) and a recognizer loss suited to tiny boxes, the Normalized Wasserstein Distance with ranking-based assignment (NWD-RKA). That paper leaves sampling untouched. No paper in the file fixes the recognizer and varies only the sampler under matched budgets for tiny-object detection.

Research Program

The shared recognizer. Every method in the study uses the same detector. It takes a fixed-size canvas, such as 512×512, and is trained with NWD-RKA, because Wang et al. show that ordinary overlap-based matching gives tiny boxes almost no positive training examples. All predictions are mapped back to original image coordinates and merged by one box-merging step, with hyperparameters tuned once on a validation split and then frozen. We train two versions of the recognizer backbone, one from supervised ImageNet weights and one from self-supervised DINO-family weights. This tests whether the raster-prior problem that SpiralFovea reports also appears in detection when glimpses are warped.

Sampling plans and cost. A sampling plan takes the full image and outputs a list of glimpses. Each glimpse is a center, a field of view, and a magnification profile. A rectangular glimpse has constant magnification. A foveal glimpse follows a magnification profile that falls off smoothly with distance from the center, as in the differentiable foveation of Schwinn et al., and is resampled onto the same fixed canvas. Because every glimpse is the same tensor size, the recognizer's cost is simply the number of glimpses times a constant. This avoids the variable-token batching problem entirely. We express the budget $b$ as the total canvas pixels the recognizer processes, divided by the native image area. The planner's own compute is added to the reported FLOPs and latency, not hidden.

The diagnostic quantity. Alongside standard AP broken out by object-size bins, we report resolution recall:

$$R_k(b) = \frac{1}{N}\sum_{i=1}^{N} \mathbf{1}\left[\max_{g \in G_b} s_i \, m_g(c_i) \ge k\right]$$

Here $N$ is the number of ground-truth objects, $G_b$ is the set of glimpses the plan emits at budget $b$, $s_i$ is object $i$'s native size in pixels, $c_i$ is its center, and $m_g(c_i)$ is the magnification that glimpse $g$ applies at that point. The quantity $m_g(c_i)$ is zero if the glimpse does not cover the object. The threshold $k$ is a minimum effective size, which we set from the recognizer's own accuracy-versus-size curve. $R_k$ measures allocation alone: it asks whether each object reached the recognizer large enough to be detectable. Pairing $R_k$ with AP separates planner failures from recognizer failures.

Oracle and naive plans. The oracle plan places the fewest glimpses that bring every ground-truth object above size $k$ within budget $b$, using a greedy set cover. It is an upper bound on allocation, not a method. The naive plans are global downsampling at budget $b$ and uniform overlapping tiles, with the downsampling factor chosen to hit budget $b$.

Our plan. Our proposed plan runs one low-resolution pass over the whole image. It then places foveal glimpses at the peaks of a combined map that sums learned density, as in SaccadeDet, with a parameter-free local-entropy map, as in SpiralFovea. A small fraction of the budget is reserved for entropy-only peaks. The intended mechanism is that an isolated tiny object may produce no density peak but still produces a local high-frequency outlier. The peripheral falloff of a foveal glimpse keeps context around the fixation. According to Harris et al., foveation rewards centering the object; we test whether that context also lowers false positives compared with a hard crop.

Baselines

The naive reference is global downsampling at each budget, which is the "do nothing clever" point. The upper reference is dense uniform tiling at budget above 1 with 25% overlap, following The Power of Tiling. That paper reports VisDrone2018 mAP rising from 11% to 36% when moving from a single frame to a 5×3 grid with a Pelee backbone. We expect a similar shape, not those exact numbers, once the recognizer is shared. Both references run through the shared recognizer, so they differ from the other methods only in sampling.

The strongest existing competitor on accuracy per unit compute is ClusDet-style cluster cropping. ClusDet reports 28.4 AP on VisDrone, against 24.4 for uniform tiling and 21.4 for Faster R-CNN, with about 40% fewer forward passes than grid tiling (ClusDet). We re-implement its cluster-and-scale proposal as a sampling plan feeding our shared recognizer, not its original DetecNet. Beating it means higher AP on the very-tiny and tiny object bins at an equal or smaller budget $b$, averaged over three seeds with non-overlapping confidence intervals. SaccadeDet's density-only planner is run the same way. Its reported 0.760 AP50 at 3.2 FPS on PANDA (SaccadeDet) is our reference point on gigapixel scenes. For single-pass recognizer quality we cite 24.7 AP for DetectoRS with NWD-RKA on AI-TOD-v2 (Wang et al.). We expect our shared recognizer at budget 1 to land near this number; a large shortfall would mean the recognizer, not the sampler, is limiting every comparison. For NWD-RKA on VisDrone2019 and for any method on DOTA-v2.0, no reported result was found on Lacuna.

The baseline we consider most dangerous is skip-empty tiling. It uses uniform tiles, but a tiny classifier on a low-resolution thumbnail discards tiles predicted empty, and the saved budget is spent on overlap or higher magnification elsewhere. Its planner gets the same parameter count and training data as ours. Every planner gets the same tuning budget of 20 validation runs.

Three ablations isolate our key ingredients. The first removes the entropy term, leaving density alone. The second replaces foveal warping with rectangular crops at identical centers and budget. The third sweeps the strength of the magnification falloff, to check whether FOVI's inverted-U appears in detection.

Why not just…?

Why not just tile everything at full resolution? Tiling covers the whole field and never misses an isolated object, which is exactly why it is our upper reference. Its weakness is cost: ClusDet found over 73% of grid chips were empty. The protocol reports tiling's full accuracy-versus-budget curve. If tiling at budget 1 already sits near the oracle and adaptive plans cannot beat it below budget 1, then adaptive sampling offers no benefit on these datasets, and we will report that.

Why not just skip the empty tiles? This is a real threat. Much of the reported advantage of ClusDet and SaccadeDet may come from simply not processing background, which a thumbnail classifier can also do without foveation or learned fixations. The decisive comparison is AP and $R_k$ on the isolated-object subset at matched budget. If skip-empty tiling matches our plan there too, our method adds nothing beyond it, and the paper becomes a benchmarking result that skip-empty tiling is the baseline to beat.

Why not just use a single-pass detector with a feature pyramid and NWD, and tune it well? In the spirit of Goyal et al., a well-tuned simple model may be enough. The protocol already contains this model: it is global downsampling at budget 1 through the NWD-trained recognizer. The oracle gap then shows whether allocation has any room to add accuracy beyond it. On AI-TOD-v2, where objects average 12.7 pixels, we expect the gap to be large; on DOTA-v2.0 we expect it to be small.

Why not just prune tokens inside a transformer? SpiralFovea points out that pruning happens after early layers have already paid for every token. More importantly for tiny objects, pruning cannot add resolution the input never contained. A pruning variant of the recognizer at budget 1 is included for completeness. Its $R_k$ equals that of global downsampling by construction, since pruning removes tokens but never magnifies any region.

Proposed Experiments

The primary benchmark is AI-TOD-v2, chosen because its corrected labels keep missing annotations from masquerading as planner errors. VisDrone2019 tests generalization to clustered urban scenes, and PANDA stresses the gigapixel regime. For each dataset we define an isolated-object subset: objects whose nearest same-image neighbor is farther than a fixed multiple of the glimpse field of view. The subset is fixed before any method is run. Budgets are swept over 0.1, 0.25, 0.5, 1, and 2. Latency is measured on one GPU at batch size 1 and batch size 8. Model selection uses validation only, never test-set peaks.

Four checks decide the outcome. The first is the oracle gap: at budget 0.25, what fraction of the AP difference between global downsampling and budget-2 tiling does the oracle recover? If the fraction is small, the study concludes that the recognizer is the bottleneck, not the sampler. The second is the skip-empty test: does our plan exceed skip-empty tiling in tiny-bin AP at the same budget, overall and on the isolated subset? The prediction is parity overall and a clear win on the isolated subset. The third is the warp test: at identical glimpse centers and budget, do foveal glimpses beat rectangular crops, and does the gain peak at moderate magnification falloff? The fourth is the pretraining test: does the warped-glimpse advantage appear only with the self-supervised backbone? If so, the raster-prior conflict from classification carries over to detection. The protocol, planners, and oracle plans will be released together, so that later adaptive detectors can be placed on the same accuracy-versus-budget curve.