Networks that must find tiny or thin things, such as a three-pixel infrared target, a traffic pole, or a vertebral pedicle edge in an X-ray, work best when they give priority to their earliest, high-resolution layers. Those layers still hold exact positions and local contrast that deeper, downsampled layers blur away. The same fields often have to train on simulated data, because real pixel-level labels are scarce. This creates a conflict nobody has measured directly. Simulators render clean high frequencies: idealized noise, no scatter, crisp rasterized edges. The early layers that shallow-priority networks depend on are exactly the layers that respond to those high frequencies. So the architectural choice that helps small structures may also route the synthetic-to-real mismatch straight into the parts of the model that matter most.
We propose to test that claim and to act on it. First, we measure layer by layer where the synthetic-to-real gap enters a shallow-priority network. Second, if the gap is concentrated in early layers, we remove it there with a light, non-adversarial loss. Using unlabeled real images, the loss matches the statistics of early-layer responses, computed separately for each spatial frequency band and conditioned on local image intensity. Deep layers are left alone, and an edge-supervision term stops the matching from simply switching off the fine-scale responses. In two sentences: shallow features are both the most useful and the most simulator-contaminated part of a small-structure model. Aligning only them, conditioned on local brightness so that signal-dependent noise such as Poisson noise is captured, should close most of the gap without the geometric damage that image translation or adversarial training can cause.
Core Claim
Our hypothesis is that for small and thin targets, the sim-to-real gap in a shallow-priority network sits mostly in the first few stages. If so, there is a simple test. Take a synthetic-trained model and swap in shallow stages trained on real data. That swap should recover most of the target-domain loss, and swapping deep stages should recover little. The method follows from the hypothesis. Its distinctive ingredient is conditioning. Real sensor corruption depends on the signal: X-ray quantum noise grows with intensity, and MRI ghosting and ringing follow edges and bright structures, as Diff5T documents for raw k-space. Plain feature normalization or adding noise to the input ignores that dependence. Matching shallow response statistics within bins of local intensity and within frequency bands does not.
The closest prior work is Shallow Features Guide Unsupervised Domain Adaptation for Semantic Segmentation at Class Boundaries. It makes shallow features domain-invariant through an auxiliary semantic-edge task, uses them to compute a smooth 2D displacement field that pulls deep features toward boundaries, and applies copy-paste augmentation with class-wise mask erosion. It motivates our work, since it shows shallow features anchor boundaries across GTA5→Cityscapes. We differ in three ways. We never warp features, so we avoid its assumption that boundaries are continuous. We do not erode masks, a step that erases objects only a few pixels wide. We target sensor statistics rather than game-graphics texture, a direction its authors leave open for medical imaging. Two other works also bridge the directions but use fixed, hand-chosen shallow cues. Semantic edge-guided 2D/3D registration for vertebrae segments expert-selected pedicle and vertebral-wall edges from simulated projections and real X-rays, then registers them with a derivative-free optimizer, and reports a residual gap from scatter. Synthetic-to-real instance segmentation of clinical cluster cells fuses shallow contours through contour attention but bridges only a cut-and-paste composition gap. Neither asks where in the network the gap lives.
Research Program
The model is a standard encoder-decoder with an explicit shallow branch. The branch uses the Laplacian and max-pool-residual "hint" operators from Lost in UNet, which feed local-contrast cues in before any downsampling, together with an early prediction head in the style of the small-ship detector of Yin et al.. We reuse this architecture as given, so the only new part is the training signal.
The diagnostic study comes first. For each stage $l$, we train a small domain probe on frozen features from matched synthetic and real crops and record how well it separates the domains. We then run the layer-swap test. For any benchmark with real labels we train a real-supervised model, and for every other benchmark a model briefly fine-tuned on a small labeled real split. We transplant its stages one by one into the synthetic-trained model, adding a short adapter where needed, and measure how much target accuracy each transplant recovers. The claim is confirmed if the shallow stages account for most of the recovery on small targets but not on large ones. If deep stages dominate instead, the premise is wrong and we report that.
The alignment loss applies to shallow stages $l \le L_s$. Let $c(x)$ assign each pixel to one of $B$ bins of local mean intensity. Let $F_{l,k}$ denote band $k$ of a Laplacian pyramid built on the stage-$l$ feature map. Let $\mu^{S}_{l,k,b}, \Sigma^{S}_{l,k,b}$ be the channel mean and covariance on synthetic images within bin $b$, and $\mu^{T}_{l,k,b}, \Sigma^{T}_{l,k,b}$ the same on unlabeled real images. Training minimizes
$$ \mathcal{L} = \mathcal{L}_{\text{task}}^{S} + \lambda \sum_{l \le L_s}\sum_{k}\sum_{b} \Big( \lVert \mu^{S}_{l,k,b}-\mu^{T}_{l,k,b}\rVert^2 + \lVert \Sigma^{S}_{l,k,b}-\Sigma^{T}_{l,k,b}\rVert_F^2 \Big) + \gamma\, \mathcal{L}_{\text{edge}}^{S}, $$
where $\mathcal{L}_{\text{edge}}^{S}$ requires the shallow branch to predict boundaries derived from the synthetic labels. Without it, the easiest way to match statistics would be to suppress high-frequency responses altogether. We use moment matching instead of a discriminator deliberately. A gradient-reversal adversary, as in DANP, pushes features to discard anything domain-specific, and at shallow layers that includes the fine edges the network needs. Moments with conditioning constrain only the noise-like part.
Proposed Experiments
We use three testbeds with different kinds of gap. The first is GTA5→Cityscapes, a photographic gap where published comparisons exist. There we report mIoU and trimap accuracy in narrow bands around boundaries, and break out thin classes such as poles and signs. The second is infrared small targets. We train on synthetic sequences in which tiny targets with idealized sensor noise are rendered into clean backgrounds, and test on real IRSTD1K, where 75% of targets cover under 0.02% of the frame. Metrics are IoU, nIoU and false-alarm rate, with NUDT-SIRST and SIRSTv2 held out as extra real test sets. The third is 2D/3D X-ray registration on the VIPS dataset. Training projections are rendered with the auto-differentiable DRR generator, which omits scatter and detector noise by construction. Metrics are success rate, projected target registration error in millimeters, and runtime. As a secondary check, we test on PETCCD cell clusters, where the gap comes from composition rather than the sensor. There we expect little benefit, which serves as a negative control.
Baselines
The naive reference in every testbed is the same shallow-priority network trained only on synthetic data. The real-supervised model supplies the ceiling. On IRSTD1K, the reported real-supervised numbers are 58.83% IoU for a vanilla U-Net and 61.01% for U-Net with the hint operators (Lost in UNet). We will report what fraction of the gap between synthetic-only training and 61.01% IoU our method closes. We will count it a success if it closes more of that gap than every other baseline without raising the false-alarm rate. Lost in UNet itself noted that false alarms can rise with shallow emphasis.
The strongest existing domain-adaptation method is the Cardace et al. pipeline (Shallow Features Guide UDA). It reports 53.5 mIoU on GTA5→Cityscapes against 52.1 for DACS and 51.5 for IAST. We rerun it with the same backbone, the same unlabeled real pool and the same tuning budget. Beating it means higher trimap accuracy in the narrowest boundary bands at equal or better overall mIoU. On VIPS, the strongest simulation-trained learner is CNN regression on DRR residuals, with 92.3% success and 0.282 mm error. The classical intensity-based method using mutual information plus gradient correlation reaches 92.7% and 0.260 mm but is much slower, at 4.71 s against 0.08 s. Beating the learner means exceeding 92.3% success at comparable speed. Matching the classical method's error would be the stronger result. No reported result was found on Lacuna for any method on the synthetic-to-real infrared protocol we construct, so all numbers in that setting come from our own runs.
Three ablations isolate the key ingredient. The first recomputes only the statistics of the normalization layers in shallow stages on real data. This is the unconditional, single-band special case of our loss. The second uses our loss without intensity conditioning. The third applies the loss to all stages instead of only shallow ones. Every arm gets the same number of unlabeled real images, the same number of training steps, and an equal hyperparameter search budget, with $\lambda$, $\gamma$, $B$ and $L_s$ counted within that budget.
Why not just…?
Why not just add realistic noise to the synthetic images? This is the most serious threat. Real noise residuals can be taken from unlabeled real images, for example as each image minus a denoised copy, and added to synthetic images without touching the labels. If the gap were purely additive, this would be as good as our method and much simpler. We run it as a full baseline. Our reason to expect a gap is that transplanted residuals keep neither the dependence on local intensity nor the structure-following artifacts, such as ghosting and scatter, that the 2D/3D registration review and Diff5T identify. We also run residual transplant combined with our loss. If transplant alone matches our method on X-ray and infrared, we will say so. The contribution then reduces to the diagnostic plus a recommendation of the simpler recipe.
Why not just recalibrate the normalization statistics of the early layers on real data? This is our first ablation, and it costs almost nothing. If it recovers most of the gap, intensity conditioning and frequency bands are unnecessary. We expect it to fall short specifically on low-intensity targets. A single mean and variance per channel mixes bright-region and dark-region noise together, and that dependence on brightness is exactly what signal-dependent noise produces.
Why not just translate synthetic images to look real, or use adversarial alignment? Image-to-image translation can move boundaries and invent structure. INNReg was designed because GAN translation distorts anatomy, and it reports 89.889% against 88.716% normalized cross-correlation for RegGAN. For a three-pixel target, a one-pixel geometric shift is fatal. We run a gradient-reversal adversary at shallow layers as a further baseline. We predict it will lower boundary-band accuracy on thin classes even when overall mIoU holds, because it suppresses the edges themselves.
Why not just fine-tune on a few real labels? When real labels exist, that is the right answer. VR-Caps shows that simulated pre-training followed by real fine-tuning raises polyp F1 from 0.49 to 0.70. Our setting assumes only unlabeled real data. We report the labeled-fine-tuning curve as a reference, and ask how many real labels our unlabeled method is worth.
Decisive Checks
The idea stands or falls on three results. First, the layer-swap test must show that shallow stages carry most of the recoverable gap for small targets, and much less for large ones. Second, the intensity-conditioned loss must beat both normalization recalibration and residual transplant on the infrared and X-ray testbeds, where the gap comes from the sensor. Third, the PETCCD control must show no comparable gain, which would confirm that the method addresses sensor statistics rather than acting as a generic regularizer. If the first check fails, we publish the negative diagnostic. If only the second fails, we publish the simpler recipe that won.
References
- Diff5T: Benchmarking human brain diffusion MRI with an extensive 5.0 Tesla k-space and spatial dataset
- Shallow Features Guide Unsupervised Domain Adaptation for Semantic Segmentation at Class Boundaries
- Semantic edge-guided 2D/3D registration for vertebrae in radiographs
- Synthetic-to-real: instance segmentation of clinical cluster cells with unlabeled synthetic training
- Lost in UNet: Improving Infrared Small Target Detection by Underappreciated Local Features
- High-Order Spatial Interactions Enhanced Lightweight Model for Optical Remote Sensing Image-Based Small Ship Detection
- Data-driven simulator for mechanical circulatory support with domain adversarial neural process
- Fast Auto-Differentiable Digitally Reconstructed Radiographs for Solving Inverse Problems in Intraoperative Imaging
- Real-time 2D/3D registration via CNN regression
- The Impact of Machine Learning on 2D/3D Registration for Image-guided\n Interventions: A Systematic Review and Perspective
- Unsupervised Multi-modal Medical Image Registration via Invertible Translation
- VR-Caps: A Virtual Environment for Capsule Endoscopy