Keep the Skeleton, Resample the Clutter: Diffusion Counterfactuals for Tiny Object Detection

Detectors lose tiny objects, meaning targets of a few to sixteen pixels such as distant vehicles in aerial photos or hotspots in infrared frames, for two reasons. First, the network shrinks its feature maps as it goes deeper, which dilutes a faint target. Second, the fine background detail that shares those shrunken feature cells looks, to the detector, a lot like the target. We propose a training-time fix for the second problem. For each training image, a diffusion model (a generator that turns random noise into images by denoising it step by step) repaints only the fine background detail. It is pinned so that the coarse, blurred version of the scene stays exactly the same, and the labeled tiny objects are pasted back untouched. The detector then sees many versions of one scene: the same layout, the same targets, and different clutter. Nothing is added at test time.

The intuition is that current methods either remove background detail or amplify all detail. Removing it throws away realistic distractors, which are exactly the patterns the detector must learn to reject. Amplifying it amplifies noise. We instead vary the distractors while keeping everything the detector should rely on fixed. This teaches it that background detail above a certain spatial frequency carries no information about where targets are. That frequency is set by the stride at which the backbone destroys tiny objects, the point where they shrink to about one feature cell. The generative tool comes from the low-frequency conditioning literature. The problem comes from the spatial degradation literature.

Core Claim

Our claim is that clutter-induced false alarms and misses on tiny objects are reduced by making the detector invariant to background detail above a chosen cutoff. We further claim that this invariance is best taught with realistic generated counterfactuals, not with smoothing or synthetic noise. The cutoff should be matched to the backbone stride at which tiny objects vanish. The measurable prediction is a lower false-alarm rate at equal detection probability on infrared benchmarks, and higher tiny-object AP on aerial benchmarks. Gains should peak when the cutoff matches the stride where targets reach about one feature cell, and shrink when the cutoff is much coarser or finer.

How it builds on and departs from prior work

The generative mechanism is borrowed directly from ILVR. At each reverse-diffusion step, ILVR replaces the low-pass content of the sample with that of a reference image, using a downsample-then-upsample filter with factor $N$. A small $N$ forces pixel fidelity to the reference, and a large $N$ keeps only coarse layout. Towards Unsupervised Blind Face Restoration Using Diffusion Prior supplies the recipe of running such a constrained diffusion offline to create training data for a fast feed-forward model. With this recipe, fine-tuned CodeFormer reached FID 84.65 on Wider-Test-200, against 87.60 for the original, at about 0.027 s per image. We reuse that offline pattern. Our target differs: we produce clutter variants for a detector, not clean restoration targets. Our constraint also differs, because it varies over space. The low-frequency anchor governs the whole scene, while labeled target regions are exempt and copied verbatim. DiffuseHigh and HiFlow use the same low-frequency anchoring to keep structure intact when generating 4K images. They serve human perceptual quality and never touch a downstream detector. They also constrain us: DiffuseHigh's "blurry prior" finding warns that interpolated anchors can pull samples toward blur. If the resampled background came out blurrier than real data, our method would collapse into background smoothing. We measure this directly.

On the degradation side, the motivating evidence is SET. SET found that suppressing high frequencies in the background helps tiny objects, the opposite of the effect on large objects. It acts on this with ground-truth-masked background smoothing applied to features at training time only. We keep SET's training-only, zero-inference-cost property and its use of masks. We replace suppression with resampling. Our argument is that smoothing teaches the detector that the background is smooth, which is false at test time, while resampling teaches that background detail is irrelevant.

HS-FPN takes the opposite route. It high-pass filters the lateral pyramid features and reports AP 23.6 against 20.2 for Cascade R-CNN R50 on AI-TOD. Its authors note sensitivity to the cutoff and amplification of background noise, which is the failure our method targets. DPNet and LIP change how the backbone downsamples. They are complementary architectural fixes and we do not compete with them. Task-Driven Super Resolution shows that restoration run as a separate stage serves detectors poorly (22.9 vs 37.5 mAP at 8× on VOC2007). That result argues against spending diffusion compute at test time and for spending it at training time.

Research Program

We first fine-tune an unconditional diffusion model on the unlabeled training images of each domain: AI-TOD aerial tiles, and NUDT-SIRST plus MDvsFA infrared frames. Labels are not needed, and this avoids the domain-specific-prior limitation the file notes for face methods.

For a training image $x$ with tiny-object mask $M$ (boxes dilated by a few pixels), we run ILVR-style sampling with filter $\phi_N$. We then composite

$$\tilde{x} = M \odot x + (1-M) \odot s, \qquad \phi_N(s) \approx \phi_N(x),$$

where $s$ is the generated sample whose low-pass content, at downsampling factor $N$, matches the original, and $\odot$ is elementwise product. We blend the mask boundary in the low band only, so that no seam of new high-frequency detail appears around targets. For each image we generate $k$ variants offline.

The cutoff $N$ is the key design variable. We tie it to the backbone stride $s^\*$ at which the median labeled object shrinks to about one cell. For AI-TOD's roughly 12.8-pixel mean size, that is stride 8 to 16.

The obvious failure mode is that the generator paints new, unlabeled vehicles or hotspots into the background, which becomes label noise. We filter these with a teacher detector trained on the original data. Any variant with a confident detection outside $M$ is either regenerated or has that region reverted to the original.

The detector trains on a mixture of original and counterfactual images, with the total number of iterations held fixed. On top of the standard loss, we optionally add a consistency term that asks the features inside target regions to match between $x$ and $\tilde{x}$. This isolates whether the benefit comes from data diversity alone or from explicit invariance.

We then characterize three things. The first is where in the frequency spectrum the detector's false alarms originate: we compute a wavelet energy profile of false-positive patches, with and without training on counterfactuals. The second is whether generated clutter is statistically close to real clutter: we compare high-band energy histograms, which also checks for the blurry-prior collapse. The third is how the optimal $N$ tracks $s^\*$ when the backbone stride configuration is changed.

Baselines

The naive reference point is the same detector trained on the original data with standard flips and crops, for the same iterations. We use Cascade R-CNN R50-FPN and DINO-5scale on AI-TOD, and the DNA-Net architecture on NUDT-SIRST and MDvsFA. It fixes what "no intervention" means under our own training recipe, and we report it alongside the published figure of 20.2 AP for Cascade R-CNN R50-FPN on AI-TOD from HS-FPN.

The strongest existing competitor is SET. It shares our regime exactly: it uses ground-truth masks during training only and adds zero inference cost. It reports 26.6% AP on AI-TOD with DINO-5scale and a 3.2 AP gain over RFLA. We re-run SET's released configuration, or a faithful reimplementation, under our data split, schedule, and seeds. We give its smoothing strength the same tuning budget as our cutoff $N$, a small grid on the validation set. Beating it means exceeding our reproduced SET number by more than the seed-to-seed standard deviation over three seeds. We also ask for the published 26.6 to be matched or exceeded in the same DINO-5scale setting. We further test SET combined with our method, because the two could be complementary: one acts on features, the other on data.

On infrared, the reference is DNA-Net, which reports on NUDT-SIRST an IoU of 87.09, a detection probability of 98.73%, and a false-alarm rate of 4.223×10⁻⁶. Because detection probability is already near ceiling, success there means a clearly lower false-alarm rate at matched detection probability. We read this off the operating curve, not at a single threshold. On MDvsFA, the only reported number in the file is AEDNet's (IoU 90.10, false-alarm rate 6.25×10⁻⁶). We treat it as context and do not reproduce it.

On AI-TOD with Cascade R-CNN, we include HS-FPN at its reported 23.6 AP as the architectural high-pass alternative. For AI-TOD-v2 the file reports NWD-RKA at 24.7 AP with DetectoRS. We apply our augmentation on top of NWD-RKA rather than against it, since it changes label assignment, not data.

Three ablations isolate the key ingredient. The first is the same pipeline with no low-frequency anchor, meaning unconstrained generation of the background. It tests whether the anchor matters, and we predict it hurts because layout context gets scrambled. The second replaces the diffusion sample in the background with the original low band plus random high-band noise of matched power spectrum. The third replaces it with the low band alone, which is pixel-space background smoothing. Together these separate "realistic resampling" from "any resampling" and from "suppression."

Why not just…?

Why not just smooth the background, as SET does? Smoothing is cheaper and already works, so this is the most serious alternative. Our reason to expect more is that at test time the background is not smooth. A detector trained on smoothed backgrounds sees a shift in the input distribution exactly in the band where false alarms arise. The low-band-only ablation and the reproduced SET baseline settle this directly. If they match our method in false-alarm rate on NUDT-SIRST and in AP on AI-TOD, the conclusion is that suppression suffices and the diffusion machinery is not justified. We would report that as the finding.

Why not just randomize background high frequencies with filtered noise, with no diffusion model? This is a real threat. It keeps the same anchor and the same invariance target at negligible cost. The case for diffusion is that dangerous clutter is structured: glints, wave crests, and cloud edges resemble targets, while spectrally matched noise does not. The matched-noise ablation tests this. The wavelet profile of false positives tells us whether diffusion variants specifically reduce the target-like false alarms that noise variants leave behind. If noise matches diffusion, we would recommend the noise version and drop the generator.

Why not just use copy-paste or context-aware resizing augmentation? ContextMix resizes whole images into boxes and beats CutMix on small industrial defects (Macro F1 0.948 vs 0.901). Augmentations of this kind change where targets appear or how large they are. They leave the background's clutter statistics unchanged relative to the target, which is the variable we manipulate. We include a copy-paste baseline with the same number of extra training views, so the comparison controls for data volume.

Why not just upscale inputs, or super-resolve them at test time? DPNet shows that brute-force upscaling adds substantial compute and extra negatives. The face-restoration study reports test-time diffusion at 4.0 to 15.9 s per image. Our method spends its diffusion budget once, offline. We report the offline generation cost per training image, so readers can weigh it against these test-time costs.

Proposed Experiments and Decisive Checks

The main experiments train each detector family on AI-TOD, AI-TOD-v2, VisDrone, NUDT-SIRST, and MDvsFA under three regimes: original data, SET, and our counterfactuals. We use three seeds each. We report AP together with its very-tiny and tiny breakdowns, and IoU, nIoU, detection probability, and false-alarm rate for the infrared sets. For VisDrone the file has no reported SET or NWD number, so our own reproduced baselines are the only reference.

The first decisive check is the cutoff sweep. We vary $N$ from 2 to 32 and, separately, change the backbone's early stride. If the best $N$ does not move with $s^\*$, the stride-matching part of the claim is wrong. A flat curve would mean any mild resampling helps.

The second decisive check is the noise-versus-diffusion comparison on false-alarm rate at fixed detection probability. This is where the method's reason to exist is decided.

The third is a hallucination audit. We manually inspect a sample of variants and report how often the generator inserts object-like patterns and how often the filter catches them. We then correlate the residual rate with any AP loss on large objects.

Finally, we check that large-object AP on COCO-style splits does not drop. SET's frequency-disparity result predicts that background-band interventions are safe for small objects but could hurt large ones, whose defining detail extends above the cutoff.