Object detectors spend the same computation on every pixel. That sounds neutral, but it means the compute each object receives grows with its area. A 200-pixel-wide truck is covered by hundreds of feature-map positions, while a 10-pixel car in an aerial photo falls inside one cell of a coarse grid, or less than one. The detection literature has already fixed the analogous imbalance on the training side. "Scale-equitable" label assignment, which gives every ground-truth object the same number $k$ of positive training examples regardless of size, lets a single coarse feature map compete with a full multi-resolution pyramid (You Only Look One-Level Feature). It also rescues tiny objects that ordinary box-overlap matching ignores completely (RFLA, NWD-RKA). The inference side has had no matching fix: equalized supervision still meets computation that is proportional to area.
We propose to close that gap using the main lesson from adaptive computation in language models. There, a cheap probe on early hidden states predicts how much extra compute will help a given input, and a fixed budget is spent where the predicted gain is largest (Learning How Hard to Think). Concretely, a fast single-level detector trained with scale-equitable assignment is given a fixed budget of $B$ high-resolution "refinement tiles" per image. A small probe on the shallowest feature map predicts, for each tile, the marginal detection gain from refining it, and the top-$B$ tiles are re-processed at fine stride. The claim is that allocating by predicted marginal gain behaves differently from allocating by object density or objectness. Gain should concentrate on tiny, sub-cell, ambiguous objects, and it should skip large objects that are already solved as well as specks too small to recover. Because $B$ is fixed, latency is fixed as well.
Core Claim
Under uniform per-pixel computation, the effective compute spent on object $i$ is roughly proportional to its area $a_i$. This is the inverse of what scale-equitable assignment does for supervision. We claim that a detector with a fixed total budget, redistributed across image regions by a learned marginal-gain predictor, can do two things at once. It can recover most of the small-object accuracy that single-level detectors lose relative to feature pyramids. It can also beat density-driven allocation at equal cost, especially on scenes where crowded regions contain large or easy objects.
The allocation rule is simple. Let $r$ index candidate tiles, let $\hat g(r)$ be the probe's predicted gain for tile $r$, and let $B$ be the tile budget. The detector refines
$$S = \arg\max_{|S| = B} \sum_{r \in S} \hat g(r).$$
The probe is trained to regress the measured gain $g(r) = L_{\text{coarse}}(r) - L_{\text{refined}}(r)$. Here $L$ is the detection loss on the objects assigned to tile $r$ under Gaussian receptive-field assignment, evaluated with and without the fine-stride refinement head. This mirrors the marginal-reward predictor of Damani et al., with image tiles in place of queries and refinement passes in place of samples.
Relation to Prior Work
Two single-level detectors motivate the proposal. YOLOF shows that uniform matching on one stride-32 map reaches 37.7 mAP on COCO at 32 FPS, against RetinaNet's 35.9 mAP at 13 FPS, but its small-object accuracy still trails pyramids. DFFT collapses scales into one encoder and reports 70% fewer GFLOPs. Both cut compute uniformly, which is the ceiling we try to lift: we keep their cheap substrate and add targeted, not uniform, resolution. CR-NAS reallocates backbone depth and receptive field, but it does so once, offline, for all inputs. We reallocate per image at runtime.
The closest existing work is density-adaptive aerial detection. DQ-DETR picks a global query count from a predicted object-count bin. It falls below its DINO baseline on COCO (50.2 vs 51.3 mAP), which suggests that image-level density is a weak signal when scales are mixed. Dome-DETR prunes background attention windows using a density heatmap and varies cost from 338.4 to 398.9 GFLOPs. Its authors flag the resulting latency variation as a problem for real-time use. Our proposal differs from both on three points. The allocation signal is predicted marginal gain rather than crowdedness. The unit of allocation is a local region rather than the whole image. The total budget is fixed rather than density-dependent.
The assignment method comes from RFLA and NWD-RKA, which model boxes and receptive fields as 2D Gaussians and give every object its top-$k$ nearest positions. We reuse this inside refinement tiles and to define the probe's training target. Both of those methods assume uniform compute everywhere.
From the language side, The LLM Already Knows shows that early representations separate easy from hard inputs; its probe reaches 93.09% ROC-AUC on ScienceQA. Damani et al. show that recognizing inputs where extra compute cannot help is where much of the saving comes from. We test whether both findings transfer to spatial regions of an image. One caution carries over from Early-stopping Self-Consistency: a confidently wrong signal gets locked in rather than corrected. In our setting, a probe that misses a region of tiny objects gives them no second chance. We therefore measure recall on unrefined tiles explicitly.
Research Program
The base system is a YOLOF-style detector with a dilated encoder on the stride-32 map, trained with RFLA-style Gaussian top-$k$ assignment instead of box-overlap matching. The image is divided into fixed tiles, for example 64×64 pixels on the input. A refinement head of fixed size runs on stride-8 backbone features cropped to the selected tiles. It predicts boxes for objects whose assigned positions fall in those tiles, again using Gaussian top-$k$ assignment at the finer stride. Detections from the coarse and refinement heads are merged by non-maximum suppression, the standard step that removes duplicate boxes.
The probe is a two-layer network on stride-4 or stride-8 features, pooled per tile. It costs well under 1% of backbone FLOPs.
Training has three stages. First, train the coarse detector. Second, train the refinement head on all tiles; this is affordable because it happens only at training time. Third, record the per-tile loss reduction $g(r)$ and fit the probe to it. A final joint fine-tuning stage trains the refinement head only on probe-selected tiles, so that it specializes to the distribution it will see at test time.
Two analyses accompany the system. The first is an equity analysis: for each object we log the FLOPs spent on the positions assigned to it, and we compare the spread of per-object compute across size bins under uniform, density-driven, and gain-driven allocation. The second is a characterization of what the probe learns. We regress $\hat g$ on object size, count, and local contrast, which tells us whether it amounts to a small-object detector or captures something more.
Baselines
The naive reference point is the base detector with no refinement ($B=0$), trained with Gaussian top-$k$ assignment. This measures how much a supervision fix alone achieves when tiny objects sit below the stride-32 cell size. The published anchor for this family is YOLOF's 37.7 mAP on COCO with ResNet-50. On AI-TOD-v2 we found no reported YOLOF result on Lacuna, so we will run it ourselves.
The upper-reference baseline is refinement on every tile. This is uniform high-resolution compute, close to a pyramid detector in cost. Any gain-driven configuration should approach its small-object AP at a fraction of its FLOPs.
The strongest competing allocation method is density-driven selection in the manner of Dome-DETR. We re-implement its density heatmap within our system, at the same $B$, the same backbone, the same training schedule, and the same hyperparameter search budget. This isolates the allocation signal from DETR-versus-dense-detector differences.
As published context, we report Dome-DETR-L at 34.6 AP on AI-TOD-v2 and 39.0 AP on VisDrone, DQ-DETR at 30.2 mAP on AI-TOD-v2, and NWD-RKA with DetectoRS at 24.7 AP on AI-TOD-v2. We will be clear that these systems use different detector families and training budgets, so the fair test is the matched re-implementation. Beating the density baseline means higher AP and very-tiny-object AP ($AP_{vt}$, objects under 8 pixels) at equal $B$ across a range of budgets, not only at one tuned point.
Two ablations isolate the proposal's key ingredients. The first swaps the gain probe for an objectness score, to test whether predicting gain matters beyond predicting presence. The second trains the refinement head with ordinary box-overlap assignment instead of Gaussian assignment, to test whether scale-equitable supervision is needed for refined tiles to pay off. A random-tile allocation at equal $B$ gives a floor.
Why not just…?
Why not just use a full feature pyramid? A pyramid is the uniform-compute answer, and YOLOF's comparison shows it is expensive for what it adds: RetinaNet reaches 35.9 mAP at 13 FPS against YOLOF's 37.7 at 32. In aerial images, where objects cover a small fraction of pixels, most fine-stride computation lands on empty ground. The all-tiles upper reference is our stand-in for the pyramid. If gain-driven allocation needs close to all tiles to match it, the idea fails, and the budget sweep will show that directly.
Why not just allocate by density or objectness, as Dome-DETR does? This is the real threat. Our probe might simply relearn "where small foreground is," in which case the two signals coincide and the contribution shrinks to a cleaner fixed-budget implementation. We expect them to diverge in two places. Dense clusters of large or already-confident objects score high on density but low on gain. Isolated tiny objects score low on density but high on gain. DQ-DETR's drop on COCO is indirect evidence that density misleads when scales mix. The matched comparison at equal $B$, broken down by scene type, will settle this. If gain and density allocation tie on AI-TOD-v2 and VisDrone, we will report that as a negative result.
Why not just fix supervision and skip extra compute? RFLA takes Faster R-CNN from 0.0 to 9.5 $AP_{vt}$ on AI-TOD using assignment alone. That shows supervision matters, but it runs on a multi-level backbone. On a stride-32 map, an object averaging 12.7 pixels occupies less than half a cell, and no assignment rule can restore spatial detail that the features no longer contain. The $B=0$ baseline measures exactly how far assignment alone gets in the single-level setting.
Why not just upsample the whole image or tile it uniformly at higher resolution? This is uniform allocation with a larger budget, and cost grows with image area regardless of content. It is covered by the all-tiles reference. We also add a matched-FLOPs variant that raises input resolution uniformly until its cost equals gain-driven allocation with budget $B$.
Proposed Experiments
The main benchmarks are AI-TOD-v2, for very tiny objects with the $AP_{vt}$ metric, and VisDrone, for crowded scenes with high variation in density. We also use COCO to check the failure mode seen in DQ-DETR, namely whether small-object gains cost large-object accuracy. For COCO we report $AP_S$, $AP_M$, and $AP_L$ separately.
For every method we sweep $B$ from zero to all tiles and plot AP against measured GFLOPs and against wall-clock latency on one GPU. Latency matters because DFFT's experience shows that FLOP savings do not always become speed. Its variance also shows whether the fixed budget really removes the fluctuation Dome-DETR reports.
The decisive checks are three. First, at matched budget, gain-driven allocation must beat density-driven allocation on AI-TOD-v2 AP and $AP_{vt}$, and the gap should be largest on scenes where density and predicted gain disagree most. Second, on COCO, $AP_L$ must stay within noise of the $B=0$ baseline while $AP_S$ rises. Third, the probe's predicted gain must rank tiles by realized gain well on held-out images, measured by rank correlation between $\hat g(r)$ and $g(r)$. Recall on unselected tiles must also be reported, so that confident misses of tiny-object regions are visible rather than hidden in averages.
We also report the per-object compute spread across size bins as the direct test of the equity claim. The prediction is that gain-driven allocation flattens the area-proportional curve that uniform computation produces.
References
- You Only Look One-Level Feature
- RFLA: Gaussian Receptive Field Based Label Assignment for Tiny Object Detection
- Detecting tiny objects in aerial images: A normalized Wasserstein distance and a new benchmark
- Learning How Hard to Think: Input-Adaptive Allocation of LM Computation
- Efficient Decoder-Free Object Detection with Transformers
- Computation Reallocation for Object Detection
- DQ-DETR: DETR with Dynamic Query for Tiny Object Detection
- Dome-DETR: DETR with Density-Oriented Feature-Query Manipulation for Efficient Tiny Object Detection
- The LLM Already Knows: Estimating LLM-Perceived Question Difficulty via Hidden Representations
- Escape Sky-high Cost: Early-stopping Self-Consistency for Multi-step Reasoning