Context from Noise, Evidence from Zoom: Diffusion Features for Synthetic-to-Real Tiny Object Detection

Detectors trained on rendered images, such as synthetic aerial scenes of ships, lose accuracy on real photographs. The loss is worst for tiny targets, objects only 10 to 20 pixels across. A promising fix uses diffusion models, image generators that learn by gradually removing noise from random inputs. Their internal features are unusually robust to domain shift: distilling a frozen diffusion model into an ordinary detector improves cross-domain detection by large margins (Generalized Diffusion Detector). The usual explanation is that adding noise erases rendering-specific texture and leaves shape. We argue that this explanation contains its own failure mode for tiny targets. The same noise that erases rendering texture also erases a 12-pixel boat. Separately, the latent diffusion backbones people actually use compress images 8× before any feature is computed, so such a target occupies roughly one and a half feature cells.

Our proposal splits the detector into two roles. The noisy, coarse diffusion features are good at describing the scene: water, harbor, road, beach. They also transfer across domains, so we use them to reason globally about where tiny objects can plausibly be. Clean diffusion features, computed on zoomed-in crops of those places, supply the evidence that an object is actually there. Two sentences summarize it. Domain-invariant global context and object-level evidence need different noise levels and different resolutions, and for tiny targets no single setting gives both. A detector that uses noisy full-frame features to decide where to look, and clean zoomed features to decide what is there, should transfer from simulation to reality better than any single-stream diffusion detector.

Core Claim

The claim has two parts. Only the first is a method.

The first part is an empirical regularity we call the invariance window. Let $t$ be the diffusion noise level used when extracting features. Let $t_{\text{inv}}$ be the smallest $t$ at which a linear classifier can no longer tell synthetic from real features better than a small margin above chance. Let $t_{\text{erase}}(s)$ be the smallest $t$ at which a linear probe can no longer detect objects of pixel size $s$ above a fixed fraction of its clean-image accuracy. A window exists for size $s$ when

$$ t_{\text{inv}} < t_{\text{erase}}(s), $$

meaning some noise level is domain-invariant yet still preserves the object. We predict the window is open for medium and large objects, which explains why Generalized Diffusion Detector and DiDA work on standard benchmarks. We predict it closes below roughly 20 pixels, the size at which Scale Match for Tiny Person Detection reports that standard representations break down. The prediction follows from frequency content. Noise at level $t$ suppresses spatial frequencies above a cutoff that falls as $t$ grows. Rendering artifacts and a tiny object's appearance both live above that cutoff. Scene layout lives below it.

The second part is the detector this regularity implies, described in the research program.

Prior Work and Where This Sits

Three papers motivate the proposal by revealing the gap.

Generalized Diffusion Detector distills multi-timestep features from a frozen diffusion U-Net into a Faster R-CNN student, aligning both features and proposal-level outputs. It reports a 14.0% average mAP gain across domain-generalization benchmarks. Its benchmarks contain medium and large objects, and its proposal-overlap alignment inherits the weakness of overlap-based box matching on tiny boxes.

DiDA adds forward-diffusion noise to both synthetic and real images to build an intermediate domain for semantic segmentation, reaching 76.8% mIoU on GTA5→Cityscapes with MIC. It applies noise uniformly across the image, which is exactly what destroys tiny targets.

UnityShip supplies 100,000 rendered aerial images with 194,054 ship instances. Its authors state that they did not study adaptation algorithms and that a texture and clutter gap remains.

Two papers constrain the design.

CleanDIFT shows that noise injection degrades fine detail, and that a short fine-tune yields clean, timestep-free features that work better on correspondence and depth. This tells us the local branch should be clean. It also tells us we cannot simply pick one noise level for everything.

Detecting tiny objects in aerial images: NWD-RKA shows that a 3-pixel shift drops the overlap score (IoU, intersection over union) of a 5×8 box from 0.54 to 0.14, which starves training of positive examples. It replaces overlap with a normalized Wasserstein distance (NWD) that treats each box as a 2D Gaussian. We adopt that assignment throughout so that box matching does not confound the comparison.

Three papers supply method pieces.

VPD shows that a frozen Stable Diffusion U-Net at $t=0$ gives strong dense features. GLAD shows that adding a high-resolution local crop to a global detector raises recall on 8×8-pixel drones from 0.17 to 0.51, using a hand-tuned Kalman filter and fixed crops. We make crop selection learned and context-driven. TDRG and MTTrans show that global attention helps small objects and helps domain alignment, respectively. Our global reasoning is restricted to coarse features where tiny objects are not present anyway. This sidesteps the attention-dilution problem, in which a few target pixels receive negligible weight against sky and sea.

SACNet uses diffusion only as an iterative box denoiser on real UAV data. It is related in name only.

Research Program

The program has three stages.

Stage one measures the invariance window. We extract Stable Diffusion U-Net decoder features at about ten noise levels. We do this on matched synthetic and real image sets: UnityShip versus real aerial ship datasets, and Sim10k versus Cityscapes. For each level we fit two linear probes. The first is a domain classifier on pooled features. The second is a small detection probe, trained with NWD assignment and evaluated separately by object-size bin. We also do this at controlled downsampling factors of 1×, 2×, and 4×, following the degradation protocol of RestoreDet. This lets us dial the same objects from medium to tiny and watch the window close.

This stage is cheap and decisive. If the window stays open for tiny objects, a single-stream diffusion detector at the right timestep suffices, and we would report that instead of building stage two.

Stage two builds the two-branch detector.

The global branch passes the full frame through the frozen diffusion backbone at a high noise level chosen from stage one, above $t_{\text{inv}}$. At native resolution, the 8× latent compression costs nothing here, because this branch is not asked to see the targets. A small transformer runs self-attention over these coarse tokens. It outputs a dense plausibility map, the probability that a tiny object of each class sits at each location. It is supervised from synthetic box centers.

The local branch takes the top-scoring regions, upsamples each crop by 4×, and encodes it with a CleanDIFT-style clean backbone. After zooming, a 12-pixel ship spans about six latent cells instead of one and a half. Detection queries in the local branch cross-attend to the global context tokens. Each query attends to the whole scene, but there are few queries and each sits at a candidate location, so the softmax is not dominated by background.

Context alone would miss objects in unusual places. To guard recall, a lightweight full-resolution proposal network also nominates crops. The final crop set is the union, under a fixed crop budget.

Stage three makes the system deployable and isolates the mechanism. We distill the full system into a compact convolutional student, in the spirit of Generalized Diffusion Detector. We replace its proposal-overlap alignment with NWD-matched alignment, then check whether the student keeps the gains.

Baselines

The naive reference point is a Faster R-CNN-style detector with NWD-RKA assignment, trained only on synthetic data and tested on real data. This gives the size of the gap. Training that same detector on real labels gives an oracle upper bound.

The strongest existing competitor is Generalized Diffusion Detector. It reports a 14.0% average mAP improvement over prior domain-generalization methods and 15.9% over Faster R-CNN. It is the most direct statement of "diffusion features transfer across domains," and it is what a reviewer would extend to tiny objects. We run it with its multi-timestep teacher and our NWD assignment, so the remaining difference is the architecture rather than box matching. It gets the same synthetic data, the same backbone weights, and the same tuning budget, measured in validation runs on a held-out synthetic split. Beating it means higher AP on objects under 20 pixels on real targets, with no loss on larger objects.

For the setting where unlabeled real images are available, we use MTTrans, which reports 57.9 mAP on Sim10k→Cityscapes. We add the same mean-teacher self-training to our model so the comparison is like for like. On the downsampled Sim10k→Cityscapes and UnityShip→real settings we propose, no reported result was found on Lacuna, so all baselines will be rerun by us. For in-domain tiny-object accuracy, the anchor is DetectoRS with NWD-RKA at 24.7 AP on AI-TOD-v2. We use this only to confirm that our local branch is competitive when there is no domain gap.

Two ablations isolate the key ingredient.

The first sets the global branch to $t=0$, so it uses clean features. If the noisy-context claim is right, this should hurt real-domain accuracy while leaving synthetic accuracy unchanged.

The second removes the 4× zoom and reads tiny objects directly from full-frame clean latents. This tests whether latent compression, not just domain shift, is a dominant loss.

A third ablation replaces query-to-context cross-attention with full self-attention over the frame. This measures dilution.

Why not just…?

Why not just extend the Generalized Diffusion Detector with NWD assignment? This is the real threat, and it is our main baseline. Its multi-timestep features include low-noise levels, so it might learn on its own to take context from noisy steps and detail from clean ones. Two things argue against this. First, its features are computed at a single image scale, where tiny targets fall below latent resolution. Second, if the invariance window closes, no learned mix of timesteps on the same unzoomed features recovers invariant object evidence. The stage-one measurement and the head-to-head comparison by size bin settle the question. If the gap to our method vanishes below 20 pixels, we would conclude the two-branch design is unnecessary.

Why not just tile the image at high resolution, scale-match the synthetic data, and train a standard detector? Tiling addresses resolution, and Scale Match addresses size distribution, lifting TinyPerson AP50 from 43.38% to 47.29%. Neither touches the texture and sensor gap that UnityShip reports remains after synthetic pretraining. We include a tiled, scale-matched clean-diffusion detector as a baseline. It is our method without the global branch. Its shortfall on real data measures exactly what context contributes.

Why not just translate synthetic images to look real? RetinaGAN shows that unconstrained translators distort or delete objects. Its fix depends on an existing detector that already finds the objects, which a tiny-object detector trained on synthetic data cannot provide. We do not expect translation to compete on targets of a few pixels. We still report a translation-then-detect run, with its recall on the smallest size bin, so that this is measured rather than asserted.

Proposed Experiments

The primary setting is UnityShip→real ships. We evaluate on the ship instances of DIOR, HRSC2016, LEVIR, TGRS-HRRSD, and DOTA, reporting AP by size bin and AP at NWD thresholds. We add 2× and 4× downsampled versions so that every object shrinks into the tiny regime under a controlled protocol.

The second setting is Sim10k→Cityscapes with the same downsampling. This links our results to the existing synthetic-to-real detection literature.

AI-TOD-v2 serves as an in-domain check that the local branch is a competent tiny-object detector. Cost is reported in GPU memory and frames per second for the full model and the distilled student.

The decisive checks, in order, are the following. First, does the invariance window close below about 20 pixels? This must be true before stage two is worth building. Second, does the noisy global branch beat the clean global branch on real data but not on synthetic data? This directly tests "context from noise." Third, does recall on the smallest bin stay close to that of the proposal-only union, showing that the context prior is not discarding true objects? Fourth, does the full model beat the NWD-equipped Generalized Diffusion Detector on sub-20-pixel targets, under matched data and tuning?

If the first check fails, we report the open window as the finding. If the second fails, the global branch is ordinary context modeling and the noise-level argument is wrong.