Detectors for tiny objects, such as a pedestrian eight pixels tall in a drone image, only work if they keep the image near full resolution. Every halving of resolution blurs such targets further into the background. Keeping full resolution everywhere is too expensive, so recent detectors first predict a coarse "where might objects be" mask. They then cut the high-resolution feature map into small square blocks and run the expensive layers only on the blocks the mask marks as active. This cuts arithmetic sharply on paper. On a real GPU, though, the work becomes an irregular stream of small, variably sized batches, plus copy kernels that gather blocks out of the image and scatter results back. The block size is chosen by a vision heuristic, and the kernels are vendor libraries tuned for dense, fixed shapes.
This note proposes treating the spatial block as a compiler tiling decision, chosen jointly with the GPU schedule. The gather step should be fused into the convolution's own input loads rather than run as a separate copy. The schedule should also adapt to how many blocks are active in each frame. The central claim is that the best block size, how many layers to fuse per gather, and the kernel schedule are coupled through a simple cost relation. That relation depends both on how tiny objects are spread across real aerial images and on the target GPU's on-chip memory. Searching them jointly should turn the large FLOP reductions of mask-driven detectors into wall-clock gains they currently leave unrealized, especially on edge GPUs.
Core Claim and Why It Is Not Already Done
Two systems motivate this work. SBNet groups active pixels into fixed N×N blocks, packs them into a dense batch with a custom gather kernel, runs vendor cuDNN convolutions, and scatters results back. It wraps whole residual units in one gather/scatter pair to amortize copying, and reports 1.8–2.7× wall-clock speedup over dense on ATG4D with under 1% accuracy loss. ESOD applies the same idea to tiny-object detection: an early objectness module, an adaptive feature slicer, and a head that runs only on active patches. It reports 37.9% AP on VisDrone and a GFLOP drop from 412.2 to 183.5 against YOLOv5, and it explicitly names GPU parallelization of slicing as unfinished work.
Neither system does what this proposal does. SBNet hand-picks N and treats the convolution as a black-box vendor call, which forces the packed batch to be written to and read back from global memory. ESOD reports GFLOPs and frame rate but does not tune the execution at all.
On the compiler side, the TVM-based iGPU work shows that autotuning plus custom templates for irregular vision operators such as non-maximum suppression can beat cuDNN on edge GPUs, reporting 1.62× on SqueezeNet on Jetson Nano. It evaluates only dense, low-resolution models. Our method borrows its template-plus-tuning approach and applies it to the sparse-block regime it never touched.
The constraint that shapes the design comes from TSM2X, which shows that general libraries underperform on irregular, tall-and-skinny matrix shapes. A packed batch of a few dozen small blocks is exactly that kind of shape.
The Coupling Relation
The block size matters to both communities for different reasons. Let $N$ be the block side in feature-map pixels and $d$ the number of consecutive 3×3 convolutions fused under one gather. A fused group must read an input block of side $N+2d$ so that border outputs are correct, because each 3×3 layer needs one extra pixel of context on every side. Let $A(N)$ be the number of blocks the mask marks active at size $N$; it depends on how tiny objects cluster in the image. Per-frame convolution work then scales as
$$ W(N,d) \;\propto\; A(N)\,(N+2d)^2 , $$
and the redundant fraction from overlapping borders is $1 - N^2/(N+2d)^2$.
Small $N$ tracks scattered tiny objects tightly, so $A(N)N^2$ approaches the true object area. It pays heavily in border redundancy, though, and produces blocks too small to fill a GPU's matrix units. Large $N$ wastes work on background inside each block. Larger $d$ cuts global-memory round-trips but grows the border and the on-chip footprint. The footprint must fit in shared memory or registers, the occupancy limit captured by the Real Warp Parallelism model.
The optimum therefore depends on dataset statistics and on hardware together. Neither the vision papers nor the compiler papers in the file search over it.
Research Program
We will build three components on top of an ESOD-style detector, keeping its mask predictor and detection head fixed so that only execution changes.
The first is a fused "implicit gather" convolution template for the tensor compiler. Each thread block reads its input tile directly from the original feature map, using a per-frame list of active block coordinates. It runs $d$ fused layers on-chip and writes only valid outputs back to their spatial locations. The packed intermediate batch is never materialized. The template exposes $N$, $d$, thread-block shape, register tiling, and unroll factors as tunable knobs.
The second is a joint search over block size, fusion depth, and kernel knobs. An analytical prior prunes the space before measurement, following GPURoofline for memory-versus-compute ceilings and Real Warp Parallelism for occupancy. The prior includes the $W(N,d)$ term evaluated on mask statistics from the training set. The goal is to avoid the roughly eight-hour, 1,000-trial tuning cost that the experience-guided TVM work reports for Ansor on ResNet-50 on Jetson AGX Xavier, where its analytical model reduced tuning to 600 seconds.
The third is per-frame schedule dispatch. The number of active blocks varies widely across frames, so a schedule tuned for one count is wrong for others. We will precompile a small set of schedules for buckets of active-block count and pick one at runtime from the mask, at negligible host cost.
Because $N$ also changes how much context surrounds each tiny object, accuracy is not assumed constant. We will retrain or fine-tune the sparse head at each candidate $N$ and report the joint accuracy–latency frontier, not latency alone.
Baselines
The do-nothing reference is the dense full-resolution YOLOv5 that ESOD compares against, compiled with the vendor inference stack on the same device. It anchors how much wall-clock speedup mask-driven sparsity yields at all.
The simplest resolution-preserving alternative is static image tiling from The Power of Tiling for Small Object Detection. That paper reports VisDrone2018 mAP rising from 11% to 36% with 5×3 tiling at a little over 5 FPS on Jetson TX2. We will rerun it with all tiles batched into one call rather than run sequentially, so it gets a fair hardware treatment.
The strongest existing method is ESOD as released, with its reported 37.9% AP on VisDrone and 30.9 FPS. The file does not state ESOD's hardware for that frame rate, and there is no reported result on Lacuna for ESOD on an edge GPU. We will therefore measure it ourselves on each device rather than quote the number across hardware.
The baseline most likely to close the gap is a strengthened SBNet-style pipeline built from the same detector. It keeps a separate gather, a vendor or Ansor-tuned dense convolution on the packed batch, and a separate scatter. It also gets a grid search over $N$ and padding of active-block counts to fixed buckets.
All methods share the same trained weights wherever $N$ matches, the same input resolution, and the same device and clocks. Every tuned method gets the same wall-clock tuning budget, and we will report tuning time.
Ablations isolate each ingredient: fusion depth $d=1$ (implicit gather without layer fusion), a single schedule without per-frame dispatch, and search without the analytical prior at equal tuning time. Beating the field means at least 1.3× lower median and 95th-percentile per-frame latency than the strengthened SBNet-style baseline at matched AP (within 0.5 points) on at least two of three GPUs. Failing that, it means a better accuracy–latency frontier at every operating point.
Why not just…?
Why not just grid-search the block size and keep cuDNN or TensorRT for the convolutions? This is the real threat, and it is exactly the strengthened SBNet-style baseline above. SBNet already shows that vendor dense kernels on gathered blocks give real speedups. If grid-searched $N$ with vendor kernels comes within about 10% of our fused kernels, the honest conclusion is that block size, not kernel fusion, carries the benefit, and we will report that. Our expectation that fusion matters rests on two points. The materialized packed batch adds a full global-memory write and read per fused group. And TSM2X shows vendor libraries underperform on small, irregular batch shapes. The $d=1$ ablation and the bucket-count sweep settle the question directly.
Why not just run Ansor or AutoTVM on the dense full-resolution network? That removes the gather/scatter irregularity but keeps all the background work. Background makes up most of the pixels on VisDrone according to ESOD. ESOD also reports memory growing to 161 GB when training a Vision Transformer at 1920×1920, so dense high resolution is not viable at all for attention-based detectors. Tuning a dense convolution graph can approach hardware peak, but it cannot remove work that sparsity eliminates. The dense tuned network appears in the baseline set so this is measured rather than assumed.
Why not just fix tiny-object accuracy with a better box-matching loss? Normalized Wasserstein distance with ranking-based assignment gains 4.3 AP on AI-TOD-v2 at zero runtime cost by repairing how tiny boxes are matched during training. It cannot restore pixels that downsampling has already erased, and it is orthogonal to execution speed. We will train every variant with the same loss, so any gain we report comes on top of it.
Why not just batch static tiles? Batched tiling uses dense, perfectly regular kernels and is easy to deploy. It still processes every empty tile. Its latency is set by tile count, not by how many objects are present. The batched-tiling baseline and the per-frame latency distributions will show where this holds; tiling may be competitive on dense-crowd frames, and we expect to lose there.
Proposed Experiments
We will evaluate on VisDrone, UAVDT, and TinyPerson, where ESOD's numbers give a reference point, and on AI-TOD-v2 because its mean object size of 12.7 pixels stresses small $N$. Each runs on one desktop GPU and two Jetson-class edge GPUs. Metrics are COCO-style AP and small-object AP, median and 95th-percentile per-frame latency, peak memory, tuning time, and achieved bandwidth and arithmetic throughput relative to the roofline.
The first decisive check tests the cost relation. Before any kernel tuning, does the optimum predicted from $W(N,d)$ and the occupancy limit match the measured latency optimum within one grid step across datasets and devices? If the optimal $N$ differs between VisDrone and TinyPerson, or between edge and desktop GPUs, that confirms block size cannot be fixed by a vision heuristic.
The second decisive check is the fused-versus-unfused comparison at matched $N$. It is the experiment that justifies the proposal or reduces it to "tune N."
The third tests per-frame dispatch. We will plot latency against active-block count and ask whether bucketed schedules remove the latency spikes a single static schedule shows on cluttered frames.
References
- SBNet: Sparse Blocks Network for Fast Inference
- ESOD: Efficient Small Object Detection on High-Resolution Images
- A Unified Optimization Approach for CNN Model Inference on Integrated GPUs
- TSM2X: High-performance tall-and-skinny matrix-matrix multiplication on GPUs
- A Performance Model for GPU Architectures that Considers On-Chip Resources: Application to Medical Image Registration
- GPURoofline: A Model for Guiding Performance Optimizations on GPUs
- Experience-guided, mixed-precision matrix multiplication with apache TVM for ARM processors
- The Power of Tiling for Small Object Detection
- Detecting tiny objects in aerial images: A normalized Wasserstein distance and a new benchmark