Sorting the Noise in Diffusion-Generated Detection Data

Diffusion models, which learn to turn random noise into realistic images step by step, can now produce object-detection training data in a single pass. They generate an image and draw its bounding boxes at the same time, which is attractive in fields like radiology or aerial survey where expert labels are scarce. The catch is that the boxes they emit are wrong in several different ways. Some boxes carry the wrong class. Some are misplaced. Some enclose hallucinated objects that belong to no real category at all. And some real objects the generator painted are never boxed. The standard remedy is to pass every sample through a chain of large verifier models, which brings back the cost the single-pass design was meant to avoid.

We propose to diagnose each synthetic box as clean, wrong-class, misplaced, or a non-object intruder, using only signals that are already available for free: how the generator's own box predictions change across its denoising steps, whether a box traces back to an expert-labeled seed object, and how the student detector's confidence evolves during training. Each diagnosis then gets its own fix. Wrong-class boxes are relabeled. Misplaced boxes are regenerated. Intruders become "ignore" regions, which are neither positives nor background. The central bet is that this last distinction matters in expert domains. A hallucinated blob that looks lesion-like should not be taught as background, because doing so suppresses recall on real lesions.

Core Claim

Noise in single-pass synthetic detection data is a mixture of distinct types, and in specialized domains the non-object type is large. Handling that mixture box by box, rather than image by image, recovers most of the value of a heavy verification cascade at a small fraction of its compute. The specific new element is the open-set, ignore-region path. No detection denoiser in the file separates non-object intruders from wrong-class objects, and no hybrid-noise method in the file operates on individual boxes inside a multi-object image.

Prior Work and Where This Sits

The method we build on is DiffusionEngine. It attaches a lightweight Detection-Adapter to a frozen latent diffusion model's U-Net features, so the model emits boxes during synthesis. It reports mAP gains of 3.1% on COCO, 7.6% on Pascal VOC and 11.5% on Clipart. It assumes the adapter's boxes are valid and was not tested on medical or tiny-object imagery. We keep its single-pass design and add the missing noise diagnosis.

The multi-stage alternative is InstructDET. It cascades LLaVA, LLaMA, a CLIP visual-prompt filter and BERT clustering to clean 3.6 million synthetic instructions. It supplies our heavy-verification baseline and the overhead we aim to avoid.

Three classification papers establish that different noise types need different treatment, and so motivate the decoupling:

  • EvidentialMix separates clean, wrong-class and out-of-vocabulary samples by where they fall in an evidential loss.
  • Manifold DivideMix filters out-of-distribution samples by nearest-neighbor distance in the embedding space.
  • ENCOFA treats open-set medical images as an explicit extra class.

All three decide at the level of whole images, and ENCOFA reports strong sensitivity to the assumed noise ratios. We move the decision to individual boxes. We also avoid global nearest-neighbor search, whose memory cost the Manifold DivideMix line flags.

On the detection side, two denoisers show that detectors' own training dynamics can clean labels without external models. DN-TOD tracks class-confusion across epochs, reweights samples by their confidence trends, and regenerates boxes from historical predictions. DIPS tracks average confidence and prediction fluctuation for pseudo-labels in classification. Neither handles non-object intruders or synthetic data. We reuse their dynamics signals as one input among three.

Three further papers matter in different ways:

  • Lesion-Harvester shows that about half the lesions in DeepLesion are unannotated. Penalizing them as negatives hurts, and recovering them lifts CenterNet AP from 41.0% to 48.3%. This motivates our choice to treat uncertain regions as ignored rather than as background.
  • Detecting tiny objects in aerial images shows that a 3-pixel shift drops IoU from 0.54 to 0.14. This constrains us: box-stability signals must use a scale-aware distance rather than raw IoU.
  • SimROD gets strong cross-domain gains without any generation. It is a reminder that synthesis must justify its cost.

Research Program

The first deliverable is a measurement, because the proposal only makes sense if the synthetic noise really is mixed.

Measuring the noise. We train a DiffusionEngine adapter on a small expert seed in each of three domains:

  • aerial tiny objects (AI-TOD-v2);
  • CT lesions (DeepLesion, with a seed of about 5% expert-verified images, matching Lesion-Harvester's 5.3%);
  • tropical tree crowns (SelvaBox).

We also include Pascal VOC to Clipart as a general-domain control. For the medical and aerial domains, the diffusion model is first fine-tuned on unlabeled in-domain images, and that cost is counted. We then have annotators (experts for CT) audit a stratified sample of roughly 2,000 generated boxes per domain. Each box is labeled as one of five types: clean, wrong-class, misplaced, intruder, or missing (a real-looking object left without a box). The resulting noise-composition table is the first result and is publishable by itself.

Three signals per box. The diagnoser combines three signals, all obtainable without extra large-model calls.

  • Denoising-trajectory stability. We run the lightweight adapter on U-Net features at several timesteps within the same denoising pass. The expected patterns are:
  • a box that appears early and holds its class and position is likely clean;
  • a box whose class flips is likely wrong-class;
  • a box whose position drifts is likely misplaced;
  • a box that appears only late, with probability spread across classes, is likely an intruder.
  • Seed anchoring. DiffusionEngine edits a reference seed image whose expert boxes are known. We match generated boxes to seed boxes using a normalized Wasserstein distance rather than IoU, which respects tiny objects. New, unanchored boxes receive a prior toward "intruder or genuine new content."
  • Student training dynamics. We borrow two quantities: DN-TOD's per-class confusion trend, and DIPS's confidence mean and fluctuation, both computed per box as the student detector trains.

Diagnosis and fixes. A small four-way classifier over these features is trained on half of the audited boxes and tested on the other half. Its outputs then drive the fixes:

  • wrong-class boxes take the class the confusion matrix points to;
  • misplaced boxes are replaced by a fusion of trajectory and historical student predictions;
  • intruder boxes become ignore regions that contribute no classification or regression loss;
  • high-objectness unboxed regions found by the student are also marked ignore, not background.

Baselines

The naive reference trains the detector on the real seed alone with standard augmentation. A second naive reference trains on seed plus raw DiffusionEngine output with no filtering. The gap between these two tells us whether synthesis helps at all in each domain.

The simplest filter keeps adapter boxes above a confidence threshold and treats everything else as background. The threshold is tuned on the same validation split and with the same number of trials as our method. This is the cheap competitor that most directly threatens the proposal.

The strongest existing denoiser is DN-TOD, applied unchanged to seed plus synthetic data with the same detector backbone (RFLA-based on AI-TOD, CenterNet on DeepLesion). Its reported numbers on real, not synthetic, noise are a lift from 34.9 to 41.0 AP at IoU 0.5 on AI-TOD-v2.0 with 20% box noise, and a gain of 4.9 points under 40% mixed noise. No reported result found on Lacuna for DN-TOD on synthetic diffusion data or on DeepLesion. Beating it means a statistically reliable AP gain over three seeds at equal training epochs.

The heavy-verification baseline is an InstructDET-style CLIP visual-prompt filter over every synthetic box. We report both its AP and its GPU-hours. No reported result found on Lacuna for CLIP box filtering on CT or tiny aerial objects. Our target is to match its AP within one point at under a tenth of its verification compute.

In DeepLesion we also compare against Lesion-Harvester, which mines real archives instead of synthesizing. Its reported CenterNet AP of 48.3% is the bar for whether synthesis adds anything beyond harvesting.

The key ablation replaces the ignore path with background for predicted intruders, keeping everything else fixed. This isolates the proposal's main claim. Further ablations drop each of the three signal families in turn.

Why not just…?

Why not just threshold the adapter's confidence? This may be enough in the general domain, where DiffusionEngine already reports large gains with essentially no filtering. The thresholding baseline and the intruder-as-background ablation will settle it for expert domains. If thresholding matches our method on AI-TOD and DeepLesion, the proposal reduces to the noise-composition study. We will report that outcome as such. We expect it to fail specifically on lesion recall, by the Lesion-Harvester logic that object-like regions taught as background suppress true positives.

Why not just run DN-TOD on the synthetic data? This is the most serious threat, because DN-TOD already handles class shifts and misplaced boxes from training dynamics alone. What it cannot do is recognize a box that encloses no valid class. The audit tells us how large that category is. If intruders are rare in a domain, DN-TOD should tie with us there, and we will say so. The value of our method should grow with the measured intruder fraction, and plotting per-domain gain against that fraction is a direct test.

Why not just skip synthesis and use self-supervised pretraining or copy-paste? Domain self-supervised pretraining (plankton MAE) and collage mixing (SimROD) are cheap and hallucination-free. Neither teaches localization of new object configurations, but that is an empirical claim here. We include an MAE-pretrained, copy-paste-augmented detector trained on the seed. Its compute budget is set equal to our generation-plus-training budget. If it wins, synthesis is not worth its overhead in that domain.

Why not just ask a vision-language model? The file gives two reasons to doubt this: general foundation models degrade on specialized imagery (review), and cascades are costly. The CLIP baseline makes both the accuracy and the cost measurable rather than assumed.

Decisive Checks

The idea holds if three things are true:

  1. The audit shows intruders are at least a substantial minority of noisy boxes in CT and aerial domains.
  2. The four-way diagnoser beats chance-corrected agreement on held-out audited boxes, with trajectory stability contributing measurably.
  3. The ignore-versus-background ablation shows a recall gain on real lesions and tiny objects at fixed precision.

The primary metrics are AP at IoU 0.5 and AP on small objects, plus lesion sensitivity at fixed false positives per image on DeepLesion. Alongside these we report verification GPU-hours per retained box. The idea fails cleanly if thresholding or unmodified DN-TOD comes within noise of our method in every domain.