Detecting a target only a few pixels wide forces a segmentation network into two conflicting moves. It must shrink the image to understand context, which can erase the target. It must also copy early, full-resolution features forward to the output so the target's outline survives, which carries sensor noise and cloud clutter straight into the prediction. Current infrared small-target networks resolve this tension with attention weights that learn to dim "background-looking" features. We propose instead to give the network a second, cheap stream whose only job is to predict the background at every location. It does this from the surrounding pixels alone, and it is built so that it can never see the pixel it is predicting. Wherever the high-resolution features are copied forward, the network subtracts this background estimate. A tiny target cannot be predicted from its surround, so it survives the subtraction. A cloud edge or noise texture can be predicted from its surround, so it cancels.
The idea comes from speech enhancement. Systems that model the noise as explicitly as the speech recover faint speech that subtractive filters throw away. Images can borrow that idea with one change. In pictures, the natural way to make a noise model trustworthy is to restrict what it can see, and a learned noise branch with no such restriction will tend to absorb the target. The result is a learned version of the classical "predict the background, subtract it" recipe. It works at every scale where high-resolution features are routed to the decoder, runs in a single forward pass, and needs no large transformer.
Core Claim
A background estimator whose field of view excludes a central window can be trained with only the ordinary segmentation labels. If its estimate is subtracted along the skip connections (the paths that carry high-resolution encoder features directly to the decoder), the network should match or exceed the best attention-gated detectors on detection rate and should cut false alarms near bright linear clutter such as cloud edges. The essential ingredient is the blind spot, meaning the central window the estimator cannot see. Without it, a learned background branch can copy the target into the background estimate, a failure the background-reconstruction literature calls "target leakage." If the gains vanish when the blind spot is removed, the claim is wrong.
What Exists and How This Differs
Three infrared detectors already model the background explicitly, and they define both the motivation and the constraints.
Dynamic Background Reconstruction via Transformer for Infrared Small Target Detection reconstructs a target-free background with a masked autoencoder and subtracts it from the image. A masked autoencoder is a transformer that fills in hidden image patches from visible ones. Because a target split across patch boundaries leaks into the reconstruction, the method adds a separate ResNet-34 that predicts an image shift so the target lands inside one masked patch. That paper is our strongest evidence that subtraction works. It is also our clearest constraint: leakage is real, it depends on a fragile prediction of where the target is, and the ViT-Large backbone limits the method to 19.96 FPS on an RTX TITAN. Our blind-spot design masks every location at once by construction, so it needs no shift predictor. It also works at feature level on the skip paths rather than once on raw pixels.
CodeMamba learns a codebook of background patterns from video and treats poorly fitting regions as targets. Its authors report that targets adjacent to bright cloud edges get absorbed into the background codebook, and the method depends on temporal sequences. We work on single frames. The cloud-edge regime is a failure mode we predict our design handles, because a ring-shaped surround around a point next to an edge contains the edge but not the point.
AEDNet has a two-branch module in which one branch sharpens target edges with fixed directional kernels and the other keeps background features. Its feature-retention path routes unfiltered background into the decoder, and its authors acknowledge that its edge operators amplify bright cloud boundaries. Its background branch is neither trained to reconstruct anything nor restricted in what it sees.
Two further lines frame the skip-connection problem. NS-FPN purifies high-frequency features using a fixed wavelet split and Gaussian gating. Its central observation, that deep pyramids amplify high-frequency clutter, motivates this work. Its assumption, that high-frequency energy outside a low-frequency mask is noise, is the heuristic we replace with a learned, surround-based prediction. DNA-Net and ACM preserve tiny targets with dense nested skips and point-wise channel modulation, respectively, and they suppress clutter purely by multiplying features with attention weights. They are the strongest representatives of the implicit approach we are testing against.
The method-supplying ancestors are audio systems. SN-Net runs parallel speech and noise streams that exchange features, and its ablation shows roughly a 0.5 dB SDR loss when that interaction is removed. DSRNet refines predicted speech and noise spectrograms jointly to undo over-suppression. We borrow the principle that noise deserves its own predictor. We do not borrow the symmetric twin encoder, which roughly doubles compute. In images an unconstrained noise branch has no reason not to model the target too, and the blind spot is what supplies that reason.
Research Program
The target stream is a standard U-Net, which keeps the comparison clean. We use the DNA-Net backbone so that every gain is measured against an already strong detail-preserving design. Alongside it runs a lightweight background stream, about a quarter of the target stream's channel width. Its receptive field excludes a central square window of radius $r$ around each output location.
We build the blind spot in a standard way. Four directional sub-networks each see only a half-plane offset away from the center, using shifted convolutions, and their outputs are merged. As a result, no path from the input pixels inside the window reaches the output at that location. The background stream emits features at each encoder resolution.
At each skip connection, the decoder receives two inputs. The first is the target-stream feature minus a learned linear projection of the background feature. The second is the background feature itself, through a small interaction gate in the spirit of SN-Net. The gate lets the decoder reclaim target evidence the subtraction removed by mistake.
The background stream is trained with a reconstruction loss at the pixel level and does not need its own labels. Outside the ground-truth target masks, its output head must reproduce the input image, because there the image is the background. Inside the masks, no target-free truth exists, so we drop the reconstruction term there. Because of the blind spot, the stream has no means of copying the target even if it is pushed to. The total objective is
$$\mathcal{L} = \mathcal{L}_{seg}(\hat{y}, y) + \beta \, \lVert (1-y) \odot (\hat{b} - x) \rVert_1 ,$$
where $x$ is the input image, $y$ is the binary target mask, $\hat{y}$ is the predicted mask, $\hat{b}$ is the background stream's pixel-level output, $\odot$ is element-wise multiplication, $\mathcal{L}_{seg}$ is the segmentation loss used by the baseline, and $\beta$ is a single weight tuned on validation data.
The radius $r$ is the one new inductive assumption. Infrared small targets are a few to roughly ten pixels across, so a window somewhat larger than the typical target should suffice. We will sweep $r$. We will also test a multi-radius variant that runs three windows in parallel and lets the gate choose among them, which addresses the worry that a fixed window fails on larger targets. That is the same weakness NS-FPN's authors report for their hand-set spiral sampling parameters.
We will analyze where the gains come from in three ways. We will stratify false alarms and missed detections by each target's distance to the nearest strong image edge. We will visualize the residual feature (target-stream feature minus background projection) at each skip level. And we will measure leakage directly: on validation images, we compute how much of the target's intensity appears in $\hat{b}$ inside the mask, with and without the blind spot.
Baselines
The naive reference point is the plain U-Net backbone with ordinary skip connections and the same segmentation loss. It shows how much any clutter-handling mechanism contributes at all.
The strongest single-frame detail-preserving competitor is DNA-Net. It is the right opponent because its dense nested skips and cascaded attention are exactly the implicit, attention-gated answer to the question we ask. It reports IoU 87.09%, probability of detection 98.73%, and false-alarm rate 4.223×10⁻⁶ on NUDT-SIRST with a ResNet-18 encoder.
The strongest noise-focused competitor is NS-FPN, which reports IoU 69.29%, detection probability 95.24%, and false-alarm rate 8.58 on IRSTD-1k, and IoU 78.75% with false-alarm rate 1.60 on NUAA-SIRST. The file does not give units for these false-alarm figures, so we will rerun the method rather than compare against them directly.
The closest background-subtraction method is Dynamic Background Reconstruction. It reports F1 75.01% on SIRST and 64.10% on MFIRST, but no reported IoU on Lacuna. We will compare on F1 and on throughput. AEDNet, which reports IoU 82.77% and nIoU 85.89% on NUDT-SIRST and IoU 90.10% on MDvsFA, serves as the dual-branch reference. ACM (IoU 0.743 and nIoU 0.731 on SIRST) serves as the lighter attention-gated reference.
Cited figures come from different papers that may use different splits and thresholds. We will therefore retrain DNA-Net, NS-FPN, ACM, and our model with official code where it is available. Every model uses the same train/test splits, the same augmentation, and the same number of training epochs. Each gets an equal hyperparameter search of twenty trials on a held-out validation split, and results are averaged over three seeds. We will report parameters, FLOPs, and FPS on one GPU.
Beating DNA-Net means one of two outcomes. The first is equal or better IoU and nIoU together with a statistically reliable drop in false-alarm rate on at least three of NUDT-SIRST, IRSTD-1k, NUAA-SIRST, and MDvsFA. The second is a clear improvement in the near-edge stratum, at no more than 30% added FLOPs. If the only gains are overall IoU differences within seed variance, the result does not count.
Three ablations isolate the key ingredient. The first removes the blind spot but keeps the background stream and its reconstruction loss, giving an unconstrained noise branch in the style of SN-Net. The second keeps the blind-spot stream but feeds its output only as an extra decoder input, with no subtraction on the skips. The third keeps the architecture but drops the reconstruction loss, so the background stream is just extra capacity. All three are run at matched parameter count.
Why not just…?
Why not just add attention gates to the skip connections? That is what DNA-Net and ACM already do. Their attention multiplies features by learned weights, which can dim clutter but cannot tell clutter from target when the two share an intensity profile. The DNA-Net paper's own limitations note this, and AEDNet's authors report false alarms on bright cloud edges. Our direct test is the near-edge stratification against retrained DNA-Net at matched FLOPs. If DNA-Net's attention already removes those false alarms, our method offers nothing there, and we will report that.
Why not just add an auxiliary background-reconstruction head to a standard U-Net? This is the most serious threat. Multi-task supervision alone might teach the encoder to separate background from target, without blind spots or subtraction on the skips. The "no subtraction" and "no blind spot" ablations above test exactly this. Our prediction is that an unconstrained auxiliary head reconstructs targets along with the background. Under the masked loss nothing forbids that, and this is the leakage the background-reconstruction paper fought with its shift predictor. The leakage measurement will show whether that happens. If the auxiliary head matches our full model, we would rewrite the contribution as "a background reconstruction loss is enough," which is a simpler and still useful finding.
Why not just subtract a classical background estimate, such as a median or top-hat filter, and feed the difference to DNA-Net? The evidence in DNA-Net and ACM shows that Top-Hat and IPI fail as standalone detectors on non-stationary clutter. Whether such a filter helps as an extra input channel to a deep network is untested in the file. We will run DNA-Net with a large-window median-subtracted image as an added input. This is a fixed, non-learned blind-spot estimator, so it separates the value of excluding the center from the value of learning the surround predictor. If it matches us, the learning is not earning its keep.
Why not just tune the loss to penalize false alarms more heavily? Reweighting trades missed detections for false alarms along one curve. It does not move the curve, a trade-off DSRNet's dynamic weighting was built to manage in audio. We will sweep the false-positive weight on the plain U-Net and on DNA-Net and plot detection rate against false-alarm rate. Our method should sit above that curve, not merely at a different point on it.
Proposed Experiments
The main experiment trains all models on NUDT-SIRST, IRSTD-1k, NUAA-SIRST, and MDvsFA. We report IoU, nIoU, probability of detection, and false-alarm rate, plus F1 on SIRST and MFIRST for comparison with Dynamic Background Reconstruction.
The decisive check is the ablation trio together with the leakage measurement. If removing the blind spot neither raises target leakage into $\hat{b}$ nor lowers detection of dim targets, the central mechanism is not doing what we claim.
The second check is the near-edge stratification. We will report false alarms and missed detections binned by distance to the strongest nearby gradient, on IRSTD-1k and MDvsFA, because those are the real and synthetic sets with the heaviest clutter. This is where CodeMamba and AEDNet report failures, and where the surround-prediction argument makes its sharpest prediction.
The third check is cross-dataset transfer: train on NUDT-SIRST and test on IRSTD-1k. A background predictor that has learned "what surrounds explain" should transfer better than attention weights tuned to one clutter distribution.
Finally, we will sweep the blind-spot radius against binned target size. This shows whether a fixed window fails on large targets and whether the multi-radius variant repairs it, at what cost in FPS relative to the 19.96 FPS reported for the transformer-based reconstruction approach.
References
- Dynamic Background Reconstruction via Transformer for Infrared Small Target Detection
- CodeMamba: Shifting from Target Semantics to Self-Supervised Background Manifold Learning for Singularity Detection in Infrared Sequences
- AEDNet: Anisotropic Enhanced Dual-Branch Network for Infrared Small Target Detection
- Seeing Through the Noise: Improving Infrared Small Target Detection and Segmentation from Noise Suppression Perspective
- Dense Nested Attention Network for Infrared Small Target Detection
- Asymmetric Contextual Modulation for Infrared Small Target Detection
- Interactive Speech and Noise Modeling for Speech Enhancement
- Speech and Noise Dual-Stream Spectrogram Refine Network With Speech Distortion Loss For Robust Speech Recognition