Keep the Pixels Before They Are Lost: Early Tile Routing for Real-Time Tiny-Object Detection

A detector sees a drone that is 8 pixels wide only if the evidence for it survives the network's first few downsampling steps. Standard backbones shrink the image by 8× before any multi-scale reasoning begins. At that point an 8-pixel target is roughly one feature cell mixed with background. The best current speedup for high-resolution small-object detection, QueryDet, works in a coarse-to-fine way. It finds likely object locations on a coarse feature map and then computes expensive high-resolution features only at those locations. That design has a blind spot built in: an object missed at the coarse level can never be recovered. It also needs a special sparse-convolution library to be fast, which most embedded chips do not run well.

We propose to make the "where do I need resolution?" decision much earlier, at stride 4, where tiny objects still occupy several cells. The selector is trained to prioritize recall over precision. The chosen regions are then executed as a fixed-size batch of dense tiles through a high-resolution branch, and each tile's result is written back into the feature pyramid (the stack of feature maps at several resolutions that the detection head reads from). Every operation stays a standard dense convolution, so latency is set by a tile count $K$ chosen per device, not by scene content. The claim is concrete: for sub-20-pixel targets, moving the resolution decision before the first heavy downsampling recovers objects that coarse-query methods miss, at equal or lower wall-clock latency on commodity edge hardware.

Core Claim and What Is New

The three directions play different roles here. Spatial resolution degradation is the failure we attack. Multi-scale fusion is the place where the rescued high-resolution evidence must re-enter the detector without being corrupted. Accuracy-latency is the hard constraint that shapes the execution design.

Prior work splits cleanly along the backbone/pyramid boundary, and this is the gap.

Two works adapt downsampling inside the backbone:

  • LIP learns importance-weighted pooling. It lifts COCO small-object AP for Faster R-CNN from 21.9% to 24.0%, but it keeps a uniform grid and saves no compute.
  • Content-Adaptive Downsampling keeps high resolution only in informative regions. It depends on submanifold sparse convolutions and was never built into a detection pyramid.

Several works make the pyramid or head dynamic but leave the backbone dense and uniform:

Two works decide resolution for the whole frame at once:

  • DPNet predicts one downsampling factor per image.
  • AutoFocus predicts one resolution per frame.

A single global choice wastes compute on the large empty background of aerial and maritime scenes. AutoFocus also suffers 22–41% live accuracy loss when its predictions drift.

GLAD is the closest in spirit, since it runs a full-resolution local detector on crops. However, it is a separate detector steered by a Kalman filter and motion cues, so it misses hovering or slow targets and shares no features with the global pass.

Our proposal differs from all of these in three ways:

  1. The decision is made at stride 4, inside the backbone, before resolution is lost.
  2. It is spatial, not frame-level.
  3. It runs as batched dense tiles whose features are fused back into the shared pyramid, not as sparse kernels and not as an isolated second detector.

Research Program

Stem and selector. The detector keeps an ordinary backbone and feature pyramid, for example a RetinaNet-style detector with a ResNet-50 or MobileNetV2 backbone. It adds a cheap selector attached to the stride-4 stem output. The selector predicts a coarse map of "tiny-object likelihood" over tiles of fixed size, for instance 32×32 stem cells.

Training the selector. Supervision comes directly from ground-truth boxes below a size threshold. The loss is recall-weighted, because a missed tiny object is unrecoverable while an unnecessary tile costs only compute. At inference the top-$K$ tiles are cropped from the stem features, stacked into a batch, and passed through a high-resolution branch. This branch is a copy of stages 2–3 with the stride removed, run densely on the tile batch. Its outputs are scattered into a $P_2$-resolution map that is zero outside the selected tiles, which feeds the pyramid and head.

Latency budget. Per-frame cost has three terms:

$$T \approx T_{\text{stem}}(HW) + T_{\text{coarse}}(HW/64) + K\,T_{\text{tile}}$$

Here $H \times W$ is the input size, $T_{\text{stem}}$ is the full-resolution stem cost, $T_{\text{coarse}}$ is the ordinary stride-8-and-deeper pipeline, and $T_{\text{tile}}$ is the measured cost of one dense tile. $K$ is fixed per device, so latency is predictable. That matters for control loops, where stale information (What Truly Matters in Trajectory Prediction for Autonomous Driving?) is as damaging as a wrong answer.

Gradient isolation in the fusion. SSPNet showed a specific failure in multi-scale fusion. When pyramid levels are merged by simple addition, a tiny object that is a positive training example at a shallow level is treated as a negative at deeper levels, and gradients from the deep levels corrupt the shallow features. We adopt a light version of SSPNet's fix at the tile boundary. The high-resolution branch receives gradients only from anchors assigned to objects below the size threshold, and the coarse pathway's negatives for those objects are masked. This reuses SSPNet's diagnosis without its heavy dilated-convolution context modules or attention heatmaps.

Diagnostic study. Alongside the method, we measure a quantity the literature assumes but has not reported: per object-size bin, the fraction of ground-truth tiny objects that a linear probe can still detect at stride 4, stride 8, and at QueryDet's coarse query level. This locates where recall is actually lost.

Baselines

Naive reference. The naive reference is a RetinaNet-FPN at native resolution with no high-resolution level. Its companion is the brute-force upper reference: the same detector with a dense $P_2$ level, or with the input upscaled 2×. QueryDet reports that adding dense $P_2$ drops RetinaNet from 13.6 to 4.85 FPS. This pair brackets the accuracy we hope to approach and the latency we must avoid.

QueryDet (strongest competitor). QueryDet is the only method in the file that addresses all three directions at once, and it reports real wall-clock gains. On VisDrone it improves mAP by 2.1 and $AP_{50}$ by 3.2 over its RetinaNet baseline while running 2.3× faster than the dense high-resolution detector. On COCO it gains 2.0 $AP_S$ and holds about 14.9 FPS where the dense model falls to 4.8 FPS. No absolute VisDrone mAP for QueryDet was found on Lacuna, so we will reproduce it from scratch.

To keep the comparison fair, both methods will share the same backbone, training schedule, input resolution, and hyperparameter search budget. Latency will be measured two ways: with its spconv head on a desktop GPU, and on a Jetson-class device. If spconv is unavailable on the Jetson, we will substitute QueryDet's best dense fallback and report that choice explicitly. We beat QueryDet if we reach higher $AP_S$ or $AP_{tiny}$ at equal measured latency on the edge device, and if the size-binned recall analysis shows the gain comes from objects QueryDet's coarse level missed.

Accuracy references on tiny-object benchmarks. On TinyPerson, SSPNet reports 59.13% $AP_{50}$ against a 57.05% Faster R-CNN-FPN baseline. It serves as the dense accuracy ceiling for gradient-aware fusion, and it has no latency budget. On VisDrone, DPNet reports 28.74 mAP with a MobileNetV2 backbone at 56.08 GFLOPs. It represents frame-level adaptive downsampling, and we will compare against it at matched GFLOPs and matched latency.

Ablations isolating the key ingredient. The first ablation keeps the tile branch but moves the selector to stride 8. This isolates the core claim that decision depth matters. Two further ablations remove the gradient masking and swap recall-weighted training for standard focal-loss training of the selector.

Why not just…?

Why not just upscale the input or add a dense $P_2$? It works for accuracy, and we include it as the upper reference. The cost grows with image area, as QueryDet's 13.6→4.85 FPS drop shows. DPNet additionally argues that enlargement floods training with background negatives. Our method must land close to dense-$P_2$ accuracy at a fraction of its latency; if it cannot, it has no reason to exist.

Why not just tile the full-resolution image and run a detector on every tile? This is a real threat on edge hardware, because it uses only dense kernels, exactly as we do. Its cost scales with the number of tiles covering the whole image, whereas ours scales with a small $K$ plus a shared stem. We will run exhaustive tiling with the same detector and report the $K$ at which our method matches its recall. If we need $K$ close to full coverage on VisDrone or TinyPerson, the selector adds nothing and we will say so.

Why not just put LIP pooling into QueryDet's backbone? This composition directly addresses coarse-level misses and needs no new execution scheme. It may recover much of the recall gap. It does not reduce compute, however, and the file gives no evidence of how LIP behaves below 20 pixels. We will train LIP-backbone QueryDet as a baseline under the same budget. If it matches our recall at equal latency, the contribution shrinks to the dense-tile execution alone, and we will report it that way.

Why not just lower QueryDet's query threshold? A lower threshold raises recall only when the evidence still exists at the coarse level. The stride-4 versus stride-8 probe study tests exactly this. If tiny objects remain linearly detectable at QueryDet's query level, threshold tuning should suffice, and our early-decision claim is falsified.

Decisive Experiments

Datasets and metrics. Training and evaluation use VisDrone-DET, TinyPerson (reporting $AP_{50}$, $AP_{tiny}$, and $AP_{75}$, because Scale Match shows tiny-object AP collapses at strict IoU), and COCO $AP_S$ as a general check. ARD-MAV is used for a deployment-style test. There GLAD reports 0.80 AP at 23.6 FPS on a Jetson NX Xavier, and hovering drones probe the motion blind spot that our appearance-only selector should not have.

Latency protocol. Latency is wall-clock, measured on a desktop GPU and on a Jetson NX Xavier, using the same compiled inference runtime for every method. We report full accuracy-latency curves by sweeping $K$, rather than single operating points.

The first decisive check. Before building anything else, we run the probe study. If more than a small fraction of sub-16-pixel objects are detectable at stride 4 but not at the coarse query level, the premise holds. If not, we abandon early routing in favor of tuning QueryDet.

The second decisive check. Across the latency curve on the Jetson, does our method dominate QueryDet, LIP-QueryDet, and exhaustive tiling on $AP_{tiny}$? The stride-8 selector ablation must close most of the gap to QueryDet for the gain to be attributable to decision depth.

The third check. Removing gradient masking should hurt most on TinyPerson, where SSPNet observed the inconsistency. If it does not, we will drop the masking as unnecessary.