Keeping the Denoising Branch: Contrast-Graded Query Denoising as a Cross-Scanner Boundary Refiner for 3D Medical Segmentation

Transformer object detectors once needed hundreds of epochs to train. A large part of the fix was "query denoising": during training, the model receives deliberately corrupted copies of the true object boxes and learns to repair them. Because the corrupted copy's target is known, this gives a clean, stable training signal from the first step. Practitioners then throw this repair branch away at test time. We propose to keep it. We bring the repair task into 3D medical segmentation and give it one specific change. The corruption is applied to whole 3D lesion and organ masks, and it is made large exactly where the scan's local tissue contrast is low, which is where real models and real annotators are unsure. The result is an image-conditioned module that has learned "given a wrong 3D boundary and the scan, move the boundary to the right place." At test time we apply it iteratively on top of a strong conventional 3D segmenter.

This matters because the hard failures in volumetric medical segmentation are concentrated in the same places. Small, low-contrast structures and scans from unseen scanners both fail. On the AMOS benchmark, models tested on MRI from scanners not seen in training lose more than 20% mean Dice, and small organs like the adrenal glands and duodenum are the weakest (AMOS). A refiner trained only on its own mistakes on the source hospital learns that hospital's error patterns. A refiner trained on a wide, physically motivated family of boundary errors should cover the errors that show up after a scanner change. That is the bet.

Core Claim

Query denoising in 3D does two separable jobs, and the second is the new one. First, it stabilizes training of a query-based 3D decoder. This is the known DETR effect, carried into volumes. Second, if the injected corruption follows local image contrast, the denoising branch becomes a boundary correction operator. Applied iteratively at test time, that operator improves boundary accuracy under scanner shift more than three alternatives: a refinement stage trained on the model's own predictions, a label-only shape denoiser, or test-time normalization adaptation.

We deliberately do not claim that a query transformer should replace a dense 3D U-Net. AMOS shows a well-tuned 3D-UNet at 89.04% mean Dice on CT, beating Swin-UNetr at 86.32% (AMOS). Building yet another end-to-end 3D transformer segmenter would run straight into that result. The refiner instead sits on top of a frozen nnU-Net-style model and is judged by whether it adds value there.

Research Program

The base model is a standard 3D U-Net configured nnU-Net style. It is trained on the source domain and then frozen. Its multi-scale decoder features feed a small transformer decoder whose queries are explicit 3D geometric objects, in the spirit of DAB-DETR. DAB-DETR made 2D queries into anchor boxes and stretched each query's attention into an oval matching the box. That change cut training from 500 to 50 epochs on COCO. We extend each query to a 3D ellipsoid: a center $(x,y,z)$ and extents $(w,h,d)$. We express both in millimetres rather than voxels, so the attention footprint stays anatomically consistent when slice thickness changes between scanners. Slice thickness is one of the shifts that dense 3D models handle poorly.

Each query's cross-attention onto the feature volume is multiplied by the prior

$$ A(\mathbf{p}) \propto \exp\!\Big(-\tfrac{1}{2}\sum_{k\in\{x,y,z\}} \tfrac{(p_k-c_k)^2}{(s_k/T)^2}\Big), $$

where $\mathbf{p}$ is a voxel position in mm, $c_k$ and $s_k$ are the query's center and extent along axis $k$, and $T$ is a sharpness temperature as in DAB-DETR. Each query outputs refined anchor offsets and a mask. The mask is computed by comparing the query embedding with every voxel feature.

The query regime depends on the task. For organs, each class appears once, so queries are tied one-to-one to classes. There is then no matching problem, and denoising serves only the refinement role. For lesions, the count varies across patients. We therefore use one-to-one Hungarian matching, the procedure that pairs each prediction with one ground-truth object. DN-DETR identified the instability of this matching as a cause of slow convergence. This split is itself a control: any convergence benefit should show up mainly in the lesion setting.

Denoising queries are built from ground-truth masks, not just boxes. Each true lesion or organ is turned into a corrupted copy in two ways. Its ellipsoid center and extent are jittered, as in DN-DETR. Its surface is also displaced along the normal by a smooth random field whose local standard deviation is

$$ \sigma(\mathbf{p}) = \sigma_{\min} + (\sigma_{\max}-\sigma_{\min})\,\exp\!\big(-g(\mathbf{p})/\tau\big), $$

where $g(\mathbf{p})$ is the intensity gradient magnitude across the true boundary at surface point $\mathbf{p}$, smoothed and normalized per scan. $\sigma_{\min}$ and $\sigma_{\max}$ bound the displacement, and $\tau$ sets how fast corruption decays as contrast rises. Sharp, high-contrast edges get small perturbations and blended edges get large ones.

Following DINO, we add negative denoising queries. DINO showed that teaching the model to answer "no object" for noised queries placed far from any true object suppresses duplicates. Our negatives are perturbations displaced beyond the contrast-scaled band, and they are supervised to predict background or "no lesion."

The supervision target is softened following Bounding Maps. That work found that linear center-to-edge ramps beat hard binary boxes for low-contrast CT lesions, raising Faster R-CNN sensitivity from 82.43% to 86.28% at 4 false positives per image. We apply a narrow linear ramp across the annotated surface, so that annotator disagreement at blended edges is not fitted as hard truth.

At test time, the frozen base model's prediction initializes the queries. Each predicted organ or connected lesion component becomes an ellipsoid plus mask. The decoder then runs $K$ refinement passes, each feeding its output mask back as the next input. This is the iterative-pondering idea of RecycleNet and of the boundary-tracing agent in Deep Poincaré Map. The difference is that it operates natively in 3D, needs no manual seed, and has been trained on a controlled error distribution rather than on its own outputs. Deep Poincaré Map motivates the domain-shift hypothesis. A 2D policy that reacts to local geometry rather than global intensity transferred zero-shot from Sunnybrook to STACOM at 0.74 Dice, against prior transfer results of 0.66 and 0.51. Our refiner is also a local correction operator, but it is volumetric and learned from scalable synthetic perturbations.

Baselines

The naive reference is the frozen 3D U-Net alone, trained nnU-Net style with its standard augmentation, evaluated with no refinement. For in-domain AMOS CT, the reported anchor for this family is 89.04% mean Dice for 3D-UNet (AMOS). We retrain it under our own splits rather than borrowing the number, and report the published value only as a sanity check. It is the strongest dense competitor, which is why everything else is built on top of it.

The strongest refinement competitor is a second-stage network trained on the base model's own predictions. This follows the double encoder-decoder design of Galdrán et al., which feeds a first segmentation back with the image into a second network and was reported to stabilize predictions across unseen centers. We implement it with the same query decoder, the same parameter count, the same number of test-time passes, and the same training-step and hyperparameter-search budget. Only the source of its training inputs differs: it sees real stage-one outputs instead of synthetic contrast-graded perturbations. This is the comparison that isolates our idea. No reported result was found on Lacuna for this design on 3D CT or MRI.

The label-only shape prior baseline is a denoising autoencoder on segmentation maps in the style of SUD. SUD cleans label maps without looking at the image and reported 50–75% lower 95th-percentile Hausdorff distance (HD95) at cortical boundaries. We train it on the same source masks with the same perturbation family. It tests whether the image conditioning in our refiner matters at all.

The test-time adaptation baseline is DeTTA from Wen et al.. It updates only batch-normalization layers with a self-supervised Noise2Void denoising loss, one step per volume, and lifted CHAOS-to-LiTS liver Dice from 85.33% to 87.14% in its own 2D setting. We port it to the 3D base model with the same one-step budget. We also test it combined with our refiner, since the two are not mutually exclusive.

For lesions on AutoPET-II, the reported number to beat is VeloxSeg's 62.51% Dice (VeloxSeg). Its official code (JinPLu/VeloxSeg) integrates with nnU-Net preprocessing, so it can be rerun under our split.

Two ablations isolate the key ingredient. The first replaces contrast-graded noise with uniform noise of matched average magnitude. The second removes negative queries. A further convergence ablation drops the denoising branch entirely, which is the end-to-end 3D DETR-style decoder without DN.

Beating the strongest competitor means a statistically significant reduction in HD95 and normalized surface Dice error on held-out scanners, assessed with a paired per-case bootstrap. This must hold relative to the own-prediction refiner at equal compute, without loss of in-domain mean Dice.

Why not just…?

Why not just use nnU-Net with heavier augmentation? This is the most serious threat, because AMOS shows that well-tuned convolutional models beat 3D transformers in-domain. But the same benchmark reports a more than 20% drop on unseen MRI scanners for the models it tested, so augmentation as practiced did not close the shift gap (AMOS). We do not compete with nnU-Net; we refine it. We also include an arm that gives nnU-Net our extra training compute as additional intensity and resolution augmentation epochs. If that arm matches the refiner on held-out scanners, the proposal fails, and we will say so.

Why not just train a cascade on the model's own errors? It is simpler and is the natural refinement design. Our hypothesis is specific, though. On the source domain, a strong base model makes few, small errors, so a self-trained refiner sees a narrow error distribution. After a scanner change, error magnitudes and locations change. The decisive measurement, described below, checks whether target-domain error statistics fall inside the synthetic training distribution but outside the self-generated one, and whether refinement gains track that coverage.

Why not just adapt batch normalization at test time? DeTTA is cheap and reported gains. However, its authors note that it can over-tune and destabilize a well-calibrated model when the shift is small (Wen et al.). It also changes intensity statistics without supplying any boundary or shape correction. We expect the two to be complementary, and the combined arm tests this. If DeTTA alone matches our gains on AMOS MRI scanners, the boundary-refinement story is unnecessary for intensity-driven shift.

Why not just denoise the label maps with a shape autoencoder? SUD shows label-space denoising sharpens boundaries, but it never looks at the image. At blended edges, the correct boundary depends on faint local evidence that a shape prior cannot see. The SUD-style baseline uses identical perturbations, so the gap between it and our refiner directly measures what image conditioning contributes.

Proposed Experiments

The domain-shift experiment uses AMOS with leave-one-scanner-out evaluation. The MRI scanners carry the reported collapse, and we also hold out CT scanners. A CT-trained to MRI-tested arm addresses the open cross-modality problem. Organs are stratified into large (liver, spleen) and small low-contrast (adrenal glands, duodenum, pancreas). Metrics are Dice, HD95, and normalized surface Dice. Our prediction is that refinement gains concentrate on the small low-contrast group and on held-out scanners, and are near zero on large organs in-domain. A uniform gain everywhere would suggest a generic capacity effect rather than the proposed mechanism.

The convergence experiment uses AutoPET-II whole-body lesions, where lesion counts vary and matching instability is real. We report validation Dice and lesion-level detection sensitivity against training steps for three versions: the query decoder with no denoising, with DN-style box noise, and with contrast-graded mask noise. We also log the matching-instability measure from DN-DETR, which counts how often query-to-lesion assignments flip between epochs. The organ setting, with no matching, serves as a control. DN should help convergence much less there, and if it helps equally, the convergence mechanism is something other than matching stabilization. DeepLesion via ULS23 supplies a second lesion set for the soft-target comparison against Bounding Maps.

The decisive check is a coverage analysis. On each held-out scanner, we measure the real distribution of the base model's surface errors: signed boundary displacement as a function of local contrast. We compare it with the error distribution seen by the self-trained refiner and with our synthetic distribution, using a two-sample distance. The proposal's mechanism predicts two things. Our refiner's advantage over the own-prediction cascade should grow with the gap between source and target error distributions. It should also vanish under the uniform-noise ablation. If neither prediction holds, contrast-graded denoising is not the cause of any gain, whatever the headline numbers say.

A final sweep over the number of test-time passes $K$ checks whether boundary error keeps falling or diverges after a few passes. Divergence would cap the method's practical value.