Detectors for tiny objects face two problems at once. Their backbones shrink the image step by step, and an object 10 pixels wide becomes one or two cells on the feature map where it is finally detected. Training pipelines also stack augmentations (random changes to training images such as blur, color shifts, zooming out, and pasting objects in) that each remove some detail. Whether two augmentations hurt each other is usually treated as a fixed property of the augmentations. We argue it is not. It depends on how much of the object survives the network's downsampling, which depends on the object's size relative to the stride (the shrink factor) of the feature level that detects it, and on the downsampling layers themselves. We propose to measure augmentation damage directly in the detector's own feature space, at the pyramid level assigned to each box. We then show that this measurement predicts which single augmentations and which pairs will lower accuracy at which object sizes. Finally, we use it as a cheap per-box gate during training.
The opportunity is concrete. Two papers in this area disagree. Scale-Aware Automatic Augmentation for Object Detection found that Gaussian blending around augmented boxes helps. Augmentation for small object detection found that Gaussian blurring of pasted small-object edges hurts. Neither explains why. A damage measure tied to object size and feature stride should predict where blur turns from helpful to harmful. If it does, the same measure gives a principled replacement for the 20 GPU days of policy search that scale-aware augmentation currently costs.
Core Claim
For an object of width $w$ detected at a feature level with stride $s$, let $F_s(x)$ be the backbone's features on that object's footprint at that level, taken from a briefly trained pilot detector. We define the survival of a transform chain $T$ as
$$ \sigma(T; w, s) = \cos\big(F_s(x)|_{\text{box}},\, F_s(T(x))|_{\text{box}}\big), $$
averaged over training objects in a size bin. Here $\cos$ is cosine similarity and $|_{\text{box}}$ restricts features to the object's (geometrically aligned) footprint. Interference between transforms $T_1$ and $T_2$ is the shortfall
$$ I(T_1,T_2) = \big(1-\sigma(T_1\circ T_2)\big) - \big(1-\sigma(T_1)\big) - \big(1-\sigma(T_2)\big), $$
so positive $I$ means the composition destroys more than the two damages added together.
We make three claims. First, the drop in accuracy for each size bin after training with an augmentation is predicted by $\sigma$ better than by object size alone. Second, $I>0$ marks the pairs that the stacking literature reports as destructive, but only below a size threshold that scales with $w/s$. Third, this threshold shifts when strided downsampling is replaced by learnable downsampling, so no single hand-set pixel cutoff can be right for every backbone. If the third claim fails, the honest conclusion is that a size threshold is enough. Our experiments are designed so that this outcome is visible.
Relation to Prior Work
Interference has been documented and managed with heuristics, mostly in classification. AugMix runs augmentations in parallel chains with a consistency loss. IPMix shows that stacking image-, patch-, and pixel-level operations conflicts, and mixes in fractals instead. Selecting Data Augmentation for Simulating Interventions reports a 25.4% drop on Rotated MNIST when all augmentations are applied together, and picks a single augmentation using a domain classifier. These papers motivate our work by showing that interference is real. None of them models it, and none of them looks at localization. When Dynamic Data Selection Meets Data Augmentation gates augmentation per sample with CLIP consistency, but acknowledges that this fails in aerial and industrial domains. We avoid any external model by using the detector's own features.
On the resolution side, LIP learns importance-weighted pooling, and Texture Reconstructive Downsampling adds a back-projection branch to recover lost texture. Both were evaluated only with basic flips and crops, so their interaction with aggressive augmentation is untested. We reuse them as backbone variants, which tests whether learnable downsampling moves the survival threshold.
The connecting work is detection augmentation for small objects. SA-AutoAug searches scale-conditioned box-level policies with a Pareto scale-balance objective. Augmentation for small object detection uses copy-paste and accepts a loss on large objects. AMRNet combines adaptive cropping, mosaic, and mask resampling additively, without checking how they interact. ContextMix admits that shrinking whole images into small boxes erases tiny defects. Our closest competitor is SA-AutoAug. It conditions augmentation magnitude on object area through black-box search. We condition on measured feature survival at the assigned stride. This explains why a policy works and transfers across backbones without new search. Finally, NWD-RKA shows that IoU-based label assignment collapses under small shifts of tiny boxes. This constrains our design. An augmentation can appear harmful only because it breaks anchor assignment. We therefore train every condition with NWD-RKA, so that feature damage is separated from assignment failure.
Research Program
The work has three stages.
Pilot model and survival tables. We train a short-schedule pilot detector for each backbone: a strided ResNet-50 with a feature pyramid network, the same model with LIP pooling, and the same model with TRD. For about a dozen transforms we compute $\sigma$ over a grid of magnitudes, object-size bins, and pyramid levels. For geometric transforms, features are compared after warping the box footprint back into alignment. The result is a lookup table, and pairwise $I$ is computed for all pairs. The expected cost is a few GPU hours per backbone, because building the table needs only forward passes. This should be compared with the 20 GPU days reported for SA-AutoAug.
Interference atlas. We test whether the tables predict training outcomes. For singles and a stratified subset of pairs, we train detectors on short schedules and record the change in AP for each size bin. The main analysis is the rank correlation between predicted damage and the measured change in AP, compared against object size alone as the predictor.
Survival-gated augmentation. During full training, each box-level transform sampled for each ground-truth box is kept, weakened to the largest magnitude whose table entry exceeds a threshold $\tau$, or dropped. Pairs with $I$ above a margin are never composed on the same box. Image-level zoom-out is limited so that no object falls below the survival floor at its new stride. The threshold $\tau$ is the only tuned parameter.
Baselines
The naive reference is the detector trained with its standard flip-and-crop pipeline and NWD-RKA assignment. Next to it is the same detector with a full stacked pipeline: color jitter, blur, zoom-out, copy-paste, and mosaic applied independently. This stacked pipeline is what practitioners actually run, and it is where we expect interference to appear.
The strongest existing method for scale-aware augmentation is SA-AutoAug. It reports RetinaNet with ResNet-50 improving from 38.2% to 41.3% AP on COCO. We will reproduce that setting to verify our implementation. For aerial data, no reported result was found on Lacuna for SA-AutoAug on AI-TOD-v2 or VisDrone. We will therefore run its released policy, and where compute allows a reduced search under its scale-balance objective, on the same detectors, schedules, and data. We will also report the GPU-hours spent. For tiny-object detection, the strongest system in the file is NWD-RKA in DetectoRS, which reports 24.7 AP on AI-TOD-v2. On VisDrone, AMRNet reports 36.1 AP with a ResNeXt-101 backbone. On UAVDT, AMRNet reports an improvement from 15.2 to 18.2 AP. We compare with AMRNet's augmentation suite by swapping it in as the policy on our detector, not by citing its numbers against a different backbone.
The key ablation is a size-threshold rule: the same gate, but driven only by box width in pixels with a cutoff tuned on validation data, and with no feature measurement. A second ablation measures survival in pixel space instead of feature space, which isolates the role of the downsampling layers.
All methods get the same data, schedule, and backbone. Each gets an equal validation budget for its single key knob: $\tau$ for us, the pixel cutoff for the rule, and policy probability scaling for SA-AutoAug.
Beating the baselines means matching or exceeding SA-AutoAug's overall AP while improving small-object AP ($AP_s$, or the very-tiny and tiny bins on AI-TOD-v2) without lowering large-object AP. It also means improving on the stacked pipeline by more than seed variance across three seeds, at under a tenth of the search compute.
Why not just…?
Why not just turn off blur and photometric augmentation for boxes smaller than 16 pixels? This is the most serious threat, and it is included as an explicit ablation. Our hypothesis is that the right cutoff depends on $w/s$ and on the downsampling layer. In that case, a cutoff tuned on a strided ResNet will be wrong for the LIP and TRD variants, and wrong for pairs whose damage only appears when they are combined. We will tune the pixel rule on one backbone and transfer it to the others, and also tune it separately on each. If the separately tuned rule matches the survival gate everywhere, we will report that the measurement explains interference but is not needed to control it.
Why not just run SA-AutoAug's search? Search can find good policies, but at 20 GPU days per setting it must be repeated for every backbone and domain. It also does not explain the blur contradiction. We compare accuracy at matched compute and full-budget accuracy. A survival gate that only matches SA-AutoAug would still be useful because of the large cost reduction.
Why not just fix label assignment with NWD-RKA and keep the standard augmentations? NWD-RKA addresses only how boxes are matched to anchors and how localization error is scored. Because every condition already uses it, any remaining damage from augmentation is damage to the features themselves. If the stacked pipeline does no harm under NWD-RKA, our premise is wrong, and this is the first thing we check.
Why not just use learnable downsampling, so that features survive any augmentation? LIP raised Faster R-CNN $AP_s$ on COCO from 21.9% to 24.0%. However, it is plausible that its importance maps are themselves misled by changed textures. The three-backbone design tests directly whether learnable downsampling removes, shifts, or worsens the survival threshold.
Decisive Checks
The first check is the blur crossover. On AI-TOD-v2 and COCO, we sweep blur radius against object size for pasted and native objects, and test whether the measured sign flip in AP occurs where $\sigma$ predicts, at a fixed fraction of $w/s$. This would reconcile the SA-AutoAug and copy-paste findings.
The second is predictive power. The rank correlation between the survival table and the measured change in AP per size bin must clearly exceed that of size alone. Otherwise the core claim fails.
The third is threshold transfer. The size threshold implied by the survival measure should move between the strided, LIP, and TRD backbones, and pairs with $I>0$ should be the ones whose combined AP falls below what their individual effects predict.
The fourth is end-to-end training on AI-TOD-v2, VisDrone-DET, and COCO, judged against the baselines above, with results reported by size bin.
References
- Scale-Aware Automatic Augmentation for Object Detection
- Augmentation for small object detection
- AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty
- IPMix: Label-Preserving Data Augmentation Method for Training Robust Classifiers
- Selecting Data Augmentation for Simulating Interventions
- When Dynamic Data Selection Meets Data Augmentation: Achieving Enhanced Training Acceleration
- LIP: Local Importance-Based Pooling
- A Texture Reconstructive Downsampling for Multi-Scale Object Detection in UAV Remote-Sensing Images
- AMRNet: Chips Augmentation in Aerial Images Object Detection
- ContextMix: A context-aware data augmentation method for industrial visual inspection systems
- Detecting tiny objects in aerial images: A normalized Wasserstein distance and a new benchmark