Many of the cheapest labels in vision come from projecting a model of the world onto the image instead of from someone drawing on the image. Examples include a robot's joint readings rendered into a tool mask, a map polygon overlaid on an aerial photo, or a simulator's geometry reprojected into a real camera. These labels have the right shape but sit in the wrong place, and the error is spatially smooth. A calibration drift pushes a whole tool several pixels to the left, and a geolocation error slides a whole region. Standard pixel-wise losses punish a network that segments the true visual boundary twice, once for missing the displaced label and once for firing where the label says background. The network learns to hedge and produces blurry masks. This is why a robotic tool segmenter trained on kinematic masks reaches only 0.62 IoU in vivo against 0.72 with manual labels (Self-Supervised Surgical Tool Segmentation using Kinematic Information). It is also why larger volumes of misaligned labels can do worse than a small clean set (Weakly Labeling the Antarctic: The Penguin Colony Case).
We propose to give the segmenter a second, training-only output. It is a small head that looks at the image's coarsest, most global features together with the noisy label, and predicts a smooth, bounded displacement field. That field moves the label onto the image before the loss is computed. At test time the head is discarded. The design has two parts. A hierarchical transformer's deepest stage sees the whole scene, so it is the natural place to estimate a low-frequency, scene-wide misalignment. A convolutional decoder keeps the fine boundary work local. The payoff is an inexpensive way to train on "free but misplaced" real labels, which competes directly with training on perfectly labeled synthetic data and then crossing the simulation-to-reality gap.
Core Claim
When label error is a smooth, per-image displacement rather than random boundary jitter, the right fix is to estimate that displacement jointly with the segmenter at coarse resolution. Randomly perturbing labels or searching over rigid shifts does not model it. Concretely, let $x$ be an image and $y$ its misaligned label map. Let $f_\theta$ be the segmenter. Let $u_\phi(x, y)$ be a displacement field predicted on a coarse control grid (for example 8×8 points), smoothly upsampled and clipped to a maximum magnitude $r_{max}$. We train
$$ \mathcal{L} = \mathrm{CE}\big(f_\theta(x),\; y \circ (\mathrm{id} + u_\phi)\big) + \lambda \lVert \nabla u_\phi \rVert^2 + \mu \big\lVert \mathbb{E}_{\text{batch}}[u_\phi] - b \big\rVert^2 , $$
where $y \circ (\mathrm{id}+u_\phi)$ is the label resampled through the field, $\lambda$ penalizes non-smooth warps, and $b$ is a global bias estimated once from a small clean anchor set. The last term handles a genuine identifiability problem. A constant offset shared by every label cannot be distinguished from the network simply learning to predict shifted masks. A few dozen clean frames pin the zero point, and the learned field absorbs the per-frame drift around it.
The claim is falsifiable in a specific way. On purely rigid translation noise, this method should match a rigid shift search and gain nothing. Its advantage must appear only when the displacement varies across the image, for example two tools drifting independently, or a warp that stretches one part of a scene but not another. If the gain does not appear there, the idea is wrong.
Relation to Prior Work
Two existing mechanisms target spatial label noise, and both motivate this proposal while leaving its niche open. The shift-resilient loss of Estimating Canopy Height at Scale lets each sparse LiDAR point search a radius-$r$ neighborhood and keeps the minimum-error match. This supplies the core intuition we reuse, that the network should be scored against the best-aligned label. However, that search is per point, discrete, and rigid, and does not extend to dense masks whose error varies smoothly across the image. NSegment+ (Unlocking Robust Semantic Segmentation Performance via Label-only Elastic Deformations against Implicit Label Noise) takes a different approach. It applies random elastic warps only to labels, so the network cannot memorize shaky human boundaries, and it needs a heuristic suppression step to avoid eroding small objects. It randomizes misalignment rather than correcting it, which suits zero-mean human jitter but not a directional calibration offset. Benchmarking Label Noise in Instance Segmentation: Spatial Noise Matters constrains the design. It shows that spatial noise does not separate in the loss distribution, so the usual "drop high-loss samples" filters fail. This is why we model the displacement rather than filter examples. Its noise is also injected independently per polygon, which is why we add a correlated-drift generator below.
On the architecture side, SegFormer (SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers) supplies the encoder, a hierarchical transformer producing features at 1/4 to 1/32 resolution. UNETR (UNETR: Transformers for 3D Medical Image Segmentation) and ViT-V-Net (ViT-V-Net: Vision Transformer for Unsupervised Volumetric Medical Image Registration) supply the hybrid pattern of a transformer encoder with a convolutional decoder over skip connections. ViT-V-Net also shows that such a model can predict deformation fields. We borrow that registration capacity, but we point it at the label instead of a second image.
The sim-to-real papers define the application regime. The coronary angiography work (Automatic Translational Correction of Multi-View Coronary Angiography Based on Auto-Annotation Data Generation) avoids label noise by training on 240,000 synthetic pairs with exact labels. It then acknowledges a modality gap and a limitation to rigid translations. Rocha et al. go the other way: they train on real images with kinematic masks after a roughly 43-minute offline alignment. Our question sits between these. Is it better to have real pixels with misplaced labels, corrected on the fly, or synthetic pixels with perfect labels?
A contradiction in the file bears directly on our design. Grad et al. find that transformer-based Mask2Former tolerates boundary noise better than Mask R-CNN. By contrast, A Guide to Robust Generalization finds that CNNs such as ConvNeXt are more robust under distribution shift in low-data regimes, and Pauls et al. saw no gain from heavier models under geolocation noise. We do not assume either side. Our ablations place the warp head on a transformer's coarse stage and, separately, on a CNN encoder's coarse stage, and let the result decide.
Research Program
We would build three components.
The first is a correlated-drift noise generator. For each image it samples a smooth random displacement field from a low-resolution Gaussian grid with a controllable amplitude and spatial frequency. It adds optional per-instance independent translations to mimic separately calibrated tools, and a per-sequence bias. Unlike existing corruptions, this gives ground-truth displacement fields, so we can measure how well the head recovers them.
The second is the model itself. It uses a SegFormer MiT encoder and a lightweight convolutional decoder with skip connections from the 1/4 and 1/8 stages. The label registration head takes the 1/32 features and a downsampled one-hot label and outputs control-point displacements. Two safeguards guard against collusion between the segmenter and the warp. First, the anchor term above. Second, a periodic check on the anchor set that the segmenter's predictions, not only the warped labels, stay aligned with clean ground truth.
The third is an analysis of where label noise enters. We would probe boundary sharpness and noise-induced blur at each pyramid stage, with and without the head. This speaks to the open question of whether high-resolution skip paths carry misaligned-boundary gradients into the representation.
Baselines
The naive reference is the same SegFormer-plus-convolutional-decoder model trained with plain cross-entropy on the noisy labels. It is trained with identical data, schedule, augmentation, and number of hyperparameter trials. It sets the floor that any method must clear, and its gap to the clean-label model measures how much the noise costs. Grad et al. report the size of that cost for instance segmentation: Mask R-CNN falls from 34.6 mAP on clean COCO to 22.3 mAP under hard noise on COCO-N.
The first strong competitor is the rigid shift search of Pauls et al. adapted to masks. For each image, the loss is evaluated against the label translated by every offset within radius $r$ on a coarse integer grid, and the minimum is kept. Per-instance search is used where instance masks exist. This is the most direct competitor because it already solves the rigid-offset case with no extra parameters. On canopy height it cut global MAE to 2.43 m, against 6.47 to 6.92 m for earlier maps, though that result is on sparse points rather than dense masks. We run it with the same $r$ as our $r_{max}$ and the same backbone. Beating it means a clear mIoU and boundary-F gain on non-uniform drift, and parity on rigid drift.
The second strong competitor is NSegment+ with its scale-aware suppression, applied to the same backbone. It is the strongest published dense-mask method for boundary label noise. It reports +2.29 mIoU on ISPRS Vaihingen, +2.38 on LoveDA, and +3.39 on PASCAL VOC, averaged across models. No reported result was found on Lacuna for NSegment+ on Cityscapes or Potsdam. We tune its warp strength with the same trial budget as ours. Beating it means gains on correlated drift beyond its reported margin on implicit noise, and no loss on the Vaihingen and LoveDA settings where it was designed to work.
The third competitor is the offline-alignment route. On STRAS it is Rocha et al.'s SSTS, which reports 0.86 IoU on phantom (0.87 fully supervised) and 0.62 in vivo (0.72 fully supervised). Elsewhere it is an equivalent offline label refinement. Beating it on in vivo data means moving materially from 0.62 toward 0.72 without the offline optimization.
Two ablations isolate the key ingredient. The first swaps the warp head's input from transformer 1/32 features to the 1/32 features of a ResNet-50 U-Net, the architecture Pauls et al. used. The second varies the warp's degrees of freedom: global translation only (which should reproduce the shift search), affine, 8×8 field, and dense per-pixel field. The dense field is expected to collude and should fail the anchor check.
Why not just…?
Why not just use the rigid shift search on masks? For kinematic calibration drift on a single tool, this may be nearly all that is needed, and that is a real threat. The experiments settle it directly. The warp-degrees-of-freedom ablation includes the translation-only case, and the drift generator sweeps from purely rigid to strongly non-uniform fields. If the 8×8 field wins only when the injected noise is non-uniform, and ties otherwise, we have mapped exactly where the extra machinery pays. If it never wins, we report that the shift search is sufficient for dense masks. That would be a useful result in its own right, because nobody has tested it outside sparse points.
Why not just apply NSegment+'s label-only elastic warping? Random warps teach the network to distrust precise boundaries. That suits zero-mean human jitter, but it cannot undo a consistent offset, because the average warped label is still in the wrong place. Its own need for small-object suppression shows the cost of randomizing labels near thin structures. Our correlated-drift benchmark includes a nonzero-mean component precisely to separate "correcting" from "blurring". The prediction is that NSegment+ helps on the zero-mean component and plateaus on the biased one.
Why not just train on synthetic data with perfect labels and adapt to real images? This is the route the angiography work took, and it avoids label noise entirely. The file notes that global feature-alignment adaptation tends to smooth away fine boundaries, and that simulators miss glare, smoke, and sensor physics (VR-Caps; the X-ray review Unberath et al.). Still, whether misplaced real labels beat perfect synthetic ones is an empirical question. Our sim-to-real experiment answers it head to head with VIPER as the synthetic source, and it also tests whether combining the two helps.
Why not just use a plain CNN U-Net, since Pauls et al. saw no benefit from bigger models? This is plausible, and it is why the head is tested on both a CNN and a transformer encoder. If the CNN version performs equally well, the contribution is the label registration head rather than the hierarchical transformer, and we will say so.
Proposed Experiments
The primary testbed is Cityscapes with our correlated-drift noise, evaluated on the clean validation labels. The model is SegFormer at B2 scale for compute. For reference, SegFormer-B5 reports 84.0% mIoU on clean Cityscapes. Metrics are mIoU and boundary F-score, plus end-point error between the recovered and injected displacement fields. We also run the Cityscapes-N and COCO-N corruptions of Grad et al. to check that the method does not hurt under the stochastic noise it was not designed for.
The sim-to-real experiment trains three models and evaluates each on clean Cityscapes. The first uses only clean-label VIPER. The second uses only drift-noised Cityscapes with the registration head. The third uses both. This tests whether correctable misplaced labels on real pixels are worth more than perfect synthetic labels.
The remote sensing check uses ISPRS Vaihingen and LoveDA, whose boundary noise comes from real human annotation. It confirms the method holds up where NSegment+ is strongest.
The real-noise check uses the STRAS phantom and in vivo data, if they can be obtained. These contain genuine kinematic misalignment, and the target is the 0.62 to 0.72 IoU gap.
The decisive checks are few. The method must tie the shift search on rigid drift and beat it on non-uniform drift. The recovered fields must correlate with the injected ones. The segmenter's own predictions must stay aligned on the clean anchor set, which rules out warp-segmenter collusion. Anchor-set size will be swept down from 100 frames to zero, to show how much clean data identifiability actually requires.
References
- Self-Supervised Surgical Tool Segmentation using Kinematic Information
- Weakly Labeling the Antarctic: The Penguin Colony Case
- Estimating Canopy Height at Scale
- Unlocking Robust Semantic Segmentation Performance via Label-only Elastic Deformations against Implicit Label Noise
- Benchmarking Label Noise in Instance Segmentation: Spatial Noise Matters
- SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers
- UNETR: Transformers for 3D Medical Image Segmentation
- ViT-V-Net: Vision Transformer for Unsupervised Volumetric Medical Image Registration
- Automatic Translational Correction of Multi-View Coronary Angiography Based on Auto-Annotation Data Generation
- A Guide to Robust Generalization: The Impact of Architecture, Pre-training, and Optimization Strategy
- The Impact of Machine Learning on 2D/3D Registration for Image-guided\n Interventions: A Systematic Review and Perspective