Infrared small target detectors look for warm specks only a few pixels wide in thermal imagery. They fail in a characteristic way: they fire on clouds, sea glint and sensor noise, and they miss dim targets sitting next to bright edges. Real labelled data is scarce, so practitioners add synthetic data. That brings a second problem. Synthetic backgrounds do not look like real sensor clutter, and the generative tools used to make them look real can silently alter the very pixels that carry the label. This note proposes a generator built to preserve those pixels. It is a pixel-space diffusion model, meaning a generator that learns to turn random noise into images by gradually removing noise, trained on unlabelled real infrared frames. At each step, the network predicts the clean image directly rather than the noise. That lets a hard constraint keep the target's contrast profile fixed while the clutter around it is redrawn. One labelled image, real or synthetic, then becomes many "clutter counterfactuals": the same target at the same place and the same signal-to-clutter ratio, embedded in different realistic backgrounds.
These counterfactuals serve two purposes. They carry synthetic labels into realistic clutter, which addresses the synthetic-to-real gap. They also let us move the benefit of test-time ensembling into training. Ensembles of detectors suppress false alarms because clutter-induced errors are partly independent across members while the true target response is shared. Running several detectors, however, multiplies latency on platforms that already run near 25 frames per second. We train one detector to be consistent across clutter counterfactuals and test whether it matches a multi-model ensemble's false-alarm suppression at single-model cost.
Core Claim
A single detector trained on target-locked clutter counterfactuals should reach the false-alarm rate and detection probability of a five-member ensemble at one-fifth of the inference cost. The gain should be concentrated on two kinds of target: those adjacent to strong background edges, and those evaluated on a real domain the detector never saw labels from. The property that makes this possible is label fidelity at the pixel scale. Existing diffusion augmentation for this task does not guarantee that property, and simple copy-paste only partly provides it.
Where This Sits
The closest work is Diff-Mosaic. It stitches four training tiles, harmonizes their seams, then passes the result through a pretrained latent diffusion model to inject authentic infrared texture, blur and noise. It raises NUDT-SIRST IoU from 88.38% to 91.18%. It treats the target as part of the image to be re-rendered and works in a compressed latent space. It never checks whether a sub-10-pixel target keeps its intensity and shape after the round trip, and it produces one augmented image per mosaic rather than controlled variants of the same labelled scene. We keep the target's clean-signal profile fixed by construction and resample only the clutter.
SIRST-5K attacks false alarms from the other side. It fuses captured real sensor noise into base images and displaces or rotates targets to synthesize hard negatives. It reaches IoU 92.78, false-alarm rate 2.735 × 10⁻⁶ and detection probability 98.84. Its noise model is additive and sampled, not a learned model of structured clutter such as cloud edges. We reuse its 4,899 synthetic images as our labelled synthetic source.
Diffuse to Detect uses a physical heat-diffusion equation, not a generative model, to grow point labels into masks that do not leak across edges. It motivates our attention to edge-adjacent targets but solves a different problem: weak labels within one dataset.
The failure mode we target is documented by CodeMamba. It reconstructs the background with learned codebooks so that targets appear as residuals, and it reports that dim targets next to bright cloud boundaries get absorbed into the background. Training data that deliberately places locked targets beside freshly generated bright edges is a direct way to probe and repair this.
The method ingredient comes from the direct signal prediction literature. Human Motion Diffusion Model predicts the clean sample at every step so that geometric losses on joints and foot contact can be applied to the network output. BOOT shows that predicting the clean signal keeps a one-step student stable across all noise levels (CIFAR-10 FID 4.38 versus 4.67 for the 50-step teacher). We borrow the first idea: a constraint on the clean estimate, applied during training and sampling. Generation speed is a secondary concern here, because counterfactual generation is offline.
The ensembling side is motivated by Averaging Weights Leads to Wider Optima and Better Generalization. It shows that averaging weights along one training run approximates an ensemble at zero test-time cost on CIFAR and ImageNet. It was never tested on dense detection or across a sensor domain shift, so it is a baseline we must beat, not a component we assume works.
Research Program
The generator is a pixel-space denoising network trained on unlabelled real infrared frames from the target domain's training split, so that its notion of clutter is real. It predicts the clean image $\hat{x}_0$ at each noise level.
To make a counterfactual of a labelled image $x$ with target mask $m$, we partially noise $x$ and denoise it, regenerating structure everywhere. After each step we project the clean estimate onto the target constraint:
$$\hat{x}_0 \leftarrow (1-M)\odot \hat{x}_0 + M\odot\big(b(\hat{x}_0) + s\,p\big)$$
Here $M$ is the dilated target mask and $p$ is the target's background-subtracted intensity profile taken from the source image. $b(\hat{x}_0)$ is the local background level that the generator is currently producing, estimated from a ring around the mask. The scalar $s$ sets the signal-to-clutter ratio.
The target therefore rides additively on whatever new background the model draws. This roughly matches how thermal radiance from a small target combines with what lies behind it. Later denoising steps then harmonize blur and noise across the boundary. This is the difference from copy-paste, which drops a fixed blob onto a background it was never rendered with.
During generator training we also add a loss on $\hat{x}_0$ that penalizes contrast change inside clean training masks, following the motion-diffusion practice of applying constraints to the predicted clean sample. Varying $s$ and biasing the sampler toward bright structures near $M$ yields controlled curricula of low-contrast, edge-adjacent targets.
The detector is any standard network from the PAL codebase, with UIU-Net and DNANet as primary choices. For each labelled image we draw $K$ counterfactuals $x^{(1)},\dots,x^{(K)}$ and train with
$$\mathcal{L} = \frac{1}{K}\sum_{k=1}^{K}\mathcal{L}_{seg}\big(f(x^{(k)}),m\big) + \lambda\,\frac{1}{K}\sum_{k=1}^{K}\big\lVert f(x^{(k)})-\mathrm{sg}[\bar f]\big\rVert^2 .$$
Here $f$ is the detector's per-pixel output, $\bar f$ is the mean output across the $K$ variants, sg means the gradient is stopped, and $\lambda$ weights the consistency term. The consistency term targets the quantity an ensemble exploits: responses that change when only the clutter changes are, by construction, false alarms or misses.
Baselines
The naive reference is the same detector trained on the source labels with standard flips and crops. A second naive reference trains on the raw SIRST-5K synthetic images with no realism step. This measures the size of the synthetic-to-real gap we claim to close.
The strongest published augmentation is Diff-Mosaic, which reports 91.18% IoU and 99.47% detection probability on NUDT-SIRST. The strongest published result on that data family is SIRST-5K's 92.78 IoU at false-alarm rate 2.735 × 10⁻⁶. We rerun both augmentation pipelines with the same detector, epochs, optimizer and tuning budget as ours. Where public code exists we use it and report our reproduced numbers alongside the published ones.
The strongest inference-side competitor is an explicit ensemble: five independently seeded detectors trained on the source data with averaged probability maps, plus a variant that averages over test-time flips. A weight-averaged detector trained with a constant learning-rate tail is the zero-overhead alternative. All three receive the same total training compute as our method, counting generator training and generation.
The key ablation is a "paste counterfactual" control. The same locked target profile is added at the same SCR onto different real background crops, with no diffusion, and the same consistency loss is applied. Two further ablations isolate the remaining ingredients. One drops the consistency term while keeping the same number of generated images, unpaired; this asks whether the gain is just more data. The other uses a noise-predicting generator of matched size, applying the same projection to its derived clean estimate; this asks whether direct clean-signal prediction matters or only the constraint does.
Beating the field means two things. First, a single detector must match or exceed the five-member ensemble's false-alarm rate at equal or better detection probability. Second, it must exceed Diff-Mosaic and SIRST-5K training on the cross-domain protocols. We found no reported cross-domain or synthetic-only-to-real numbers in the file for either method, so those comparisons come from our reruns.
Why not just…?
Why not just paste synthetic blobs onto real backgrounds? This is the most serious threat. Copy-paste also locks the label and also gives real clutter. If the paste-counterfactual ablation matches our method overall and on edge-adjacent targets, the generator is unnecessary, and we will report that as the finding. Our reason to expect a gap is narrow and testable. A pasted blob next to a bright edge has a sharp, physically wrong boundary that a detector can learn as a shortcut, and the shortcut does not exist in real data. We therefore report results per bin of distance from the target to the nearest strong background gradient. Our method should win in the near-edge bins and tie elsewhere.
Why not just use Diff-Mosaic's latent diffusion augmentation? Its images look realistic, but nothing guarantees that a few-pixel target survives compression into latent space and re-rendering. If the target dims or shifts, the label becomes wrong exactly where the task is hardest. We measure this before any detector is trained. Before-and-after SCR error and centroid shift are recorded for targets passed through the Diff-Mosaic pipeline and through ours, stratified by target size. If Diff-Mosaic preserves targets just as well, this argument collapses and the comparison reduces to the consistency loss.
Why not just average weights, or run an ensemble? Weight averaging is free at test time, but its evidence comes from in-distribution classification. Averaging only helps within one basin, and it does not show the model which responses depend on clutter. A true ensemble does help, but it costs K times the latency. CodeMamba's 26.52 frames per second on a single high-end GPU shows how little headroom real-time systems have. Both are run as baselines. If weight averaging alone closes the gap to the ensemble, our ensemble-transfer claim fails, and we report only the domain-transfer benefit.
Why not a standard noise-predicting diffusion model with inpainting? The two parameterizations are interchangeable in principle. The concrete difference is that converting a noise estimate into a clean estimate amplifies errors at high noise levels, and the projection is applied to that converted estimate. The matched noise-predicting ablation settles this. If it ties, the contribution narrows to "a clean-estimate constraint," whichever way it is computed.
Proposed Experiments
We use three protocols. In the synthetic-to-real protocol, the detector trains only on the 4,899 SIRST-5K synthetic images, plus our counterfactuals of them, and is tested on NUAA-SIRST and IRSTD-1k. The generator learns clutter from unlabelled training frames of the test domain. In the real-to-real transfer protocol, training is on NUDT-SIRST and testing is on NUAA-SIRST, following the cross-dataset setup of Text-IRSTD. The in-domain protocol trains and tests on NUDT-SIRST and IRSTD-1k, for direct comparison with published numbers. Reference in-domain figures include UIU-Net's 0.7825 IoU on SIRST and Text-IRSTD's 69.57% IoU on IRSTD-1k.
Metrics are IoU, normalized IoU, detection probability and false-alarm rate. These are reported overall and in bins by local SCR and by distance to the nearest strong edge. We also report inference frames per second and memory for every system. Generation cost per counterfactual is reported for comparison with Diff-Mosaic's roughly 10 seconds per sample, though it is not part of the claim.
The decisive checks, in order, are as follows. First, the target-fidelity measurement must show that our generator preserves SCR where latent re-rendering does not. Second, the single consistency-trained detector must reach the five-member ensemble's false-alarm rate on the cross-domain protocols. Third, our method must beat the paste control in near-edge bins. A secondary probe trains a CodeMamba-style background-reconstruction detector on the same curricula, to see whether edge-adjacent absorption decreases.
If the first check fails, the idea is wrong. If the second fails but the others hold, we have a better augmentation and no ensemble result. If the third fails, the honest conclusion is that consistency training on simple pastes is enough, and diffusion is not needed for this task.
References
- Diff-Mosaic: Augmenting Realistic Representations in Infrared Small Target Detection via Diffusion Prior
- SIRST-5K: Exploring Massive Negatives Synthesis With Self-Supervised Learning for Robust Infrared Small Target Detection
- Diffuse to Detect: Bi-Level Sample Rebalancing with Pseudo-Label Diffusion for Point-Supervised Infrared Small-Target Detection
- CodeMamba: Shifting from Target Semantics to Self-Supervised Background Manifold Learning for Singularity Detection in Infrared Sequences
- Human Motion Diffusion Model
- Data-free Distillation of Diffusion Models with Bootstrapping
- Averaging Weights Leads to Wider Optima and Better Generalization
- Text-IRSTD: Leveraging Semantic Text to Promote Infrared Small Target Detection in Complex Scenes