Keeping Dice Away From the "Is There a Lesion?" Decision in Diffusion Segmentation

Diffusion models are increasingly used for medical image segmentation. A diffusion model is a generator that starts from random noise and removes it step by step; here it produces a lesion or organ mask conditioned on the scan. These models report strong overlap scores on lesion-containing benchmarks. They have never been tested on the case that matters most in clinics: a healthy scan, where the correct answer is an empty mask. Two facts suggest they will fail there. First, the most common segmentation objective adds per-pixel cross-entropy to Soft Dice, a region-overlap score. That combination makes ordinary CNNs hallucinate lesions on healthy images, because Dice becomes numerically degenerate when the true mask is empty On the Optimal Combination of Cross-Entropy and Soft Dice Losses for Lesion Segmentation with Out-of-Distribution Robustness. Second, diffusion generators settle global layout, including whether a structure exists at all, in their early, high-noise steps, and only sharpen boundaries late.

We propose to test the hypothesis that hallucinations on healthy scans are decided early in the denoising trajectory. If they are, the fix is structural rather than a matter of tuning loss weights: Dice gradients should only ever reach parameters that refine boundaries at low noise. The parameters that decide whether a lesion is present should be trained by cross-entropy alone. We build this with a small mixture of experts, meaning a set of parallel sub-networks with a gate that decides which ones run. One always-on shared expert is trained only by cross-entropy and owns the presence decision. A few boundary experts add a correction on top of it; they are gated by the diffusion timestep and by local uncertainty, and they alone receive Dice gradients. On a healthy scan the gate can switch the boundary experts off, so the empty-mask degeneracy of Dice cannot push false positives into the output.

Core Claim

Three claims follow, ordered from most to least central. The first is a diagnostic claim. Diffusion segmenters trained with a static cross-entropy-plus-Dice loss hallucinate on lesion-free images, and the hallucinated components first appear and become fixed at high-noise timesteps. We can test this by tracing the reverse process.

The second is a mechanism claim. Isolating Dice gradients to timestep-gated boundary experts, with a stop-gradient into the shared pathway, recovers the healthy-scan robustness of cross-entropy-only training while keeping the boundary quality of the hybrid loss. Prior work could only get this trade-off by switching losses across training epochs.

The third is an efficiency claim, and it is the one that justifies using routed experts rather than a dense second head. Because boundary experts are only needed late in the trajectory and only at ambiguous pixels, most of their compute can be skipped. This mirrors the sparse-early, dense-late control pattern that FlexControl found when it learned to skip control blocks in text-to-image diffusion.

The closest existing work does neighbouring things. On the Optimal Combination of Cross-Entropy and Soft Dice Losses motivates this proposal: it identifies the Dice degeneracy on negative images and fixes it by scheduling losses over training epochs in deterministic ResNets ("Hard Fine-tuning": cross-entropy for 90% of training, Dice for the last 10%). It never touches denoising timesteps or parameter isolation. Cross-Timestep and MedSegDiff-V2 supply the evidence that time-dependent conditioning matters inside diffusion segmenters. The first injects a coarse prior mask with weights that decay over timesteps; the second shifts frequency passbands as a function of timestep. Both train against a fixed objective with monolithic networks, and neither evaluates healthy scans. FlexControl supplies the routing technique: Gumbel-Sigmoid gates with a compute budget. Its gates choose between identical control blocks for image synthesis, whereas our gates choose between functionally distinct experts that are trained by different losses. Decoding Knowledge Attribution in Mixture-of-Experts supplies the design intuition: in large language model mixtures, always-on shared experts carry basic features and routed experts carry refinement. We make that split explicit and enforce it through the loss, instead of hoping it emerges.

Research Program

The base model is a conditional diffusion segmenter in the style of MedSegDiff. At each timestep $t$ the network sees the noisy mask $x_t$ and the image, and predicts a clean mask $\hat{x}_0$. We split the final decoder stage into two parts. The first is a shared expert producing logits $s$. The second is $K$ boundary experts (we plan $K=4$) producing residual logits $r_k$. Each boundary expert is weighted by a gate $g_k(t,u) \in [0,1]$, which reads the timestep embedding and a per-pixel uncertainty map $u$; $u$ is the distance of the previous step's prediction from 0.5. The predicted mask is $\hat{x}_0 = \sigma\big(s + \sum_k g_k r_k\big)$, where $\sigma$ is the logistic function.

Training adds two segmentation terms to the usual denoising loss:

$$ L = L_{\text{denoise}} + L_{\text{BCE}}\big(\hat{x}_0, y\big) + w(t)\,\mathbb{1}[\textstyle\sum_i y_i > 0]\; L_{\text{Dice}}\big(\sigma(\bar{s} + \textstyle\sum_k g_k r_k),\, y\big) + \lambda\, L_{\text{cost}}(g). $$

Here $y$ is the ground-truth mask, and $\bar{s}$ is the shared logit with its gradient stopped, so Dice cannot change the shared expert. The weight $w(t)$ is zero at high noise and positive at low noise, and $\mathbb{1}[\cdot]$ restricts Dice to images that contain foreground. $L_{\text{cost}}$ is FlexControl's squared hinge on mean gate activation above a target budget $\gamma$. We fix $w(t)$ rather than learning it, because a weight learned by minimizing the same loss would simply shrink to zero. Instead we compare a monotonic ramp against a mid-trajectory bump. The bump is motivated by the finding in Learn to Guide Your Diffusion Model that optimal guidance schedules are often non-monotonic, peaking mid-trajectory. Cross-entropy reaches every parameter, including the gates, at every timestep. This means that on healthy scans it actively trains the gates to close.

The program has three stages. The first is a trajectory diagnosis on a static-loss MedSegDiff baseline. On healthy images we record the timestep at which each false-positive connected component first exceeds threshold in $\hat{x}_0$. We then run a resampling intervention: re-draw the noise from timestep $t$ onward and measure how often the false positive survives. If components that survive resampling from mid-trajectory are mostly already present at high $t$, the early-decision hypothesis holds. The second stage trains the gated-expert model and its ablations. The third measures the compute curve: gate activation as a function of $t$, and wall-clock time per sampled mask.

Baselines

The naive reference is a MedSegDiff-style segmenter trained with the denoising loss plus a static, equal-weight cross-entropy and Dice loss at every timestep. This is what current diffusion segmenters effectively do. A second reference is the same model trained with cross-entropy only. On deterministic CNNs this is the variant that stays robust on healthy images: on DFUC 2022 with remission feet, a ResNet101 trained with cross-entropy only reached 89.33% Dice versus 87.92% for the hybrid (On the Optimal Combination…).

The strongest existing method for the healthy-scan problem is Hard Fine-tuning from the same paper, which placed 3rd in DFUC 2022 with a ResNeXt-101. We run it twice. The first run uses its original deterministic CNN, so we inherit its known behaviour. The second run transplants it into the diffusion segmenter: cross-entropy for 90% of epochs, then Dice for 10%, applied at all timesteps. This transplant is the key competitor, because it asks whether epoch scheduling alone already fixes the diffusion model. For in-distribution boundary quality, the strongest diffusion references are MedSegDiff, which reports 89.9% Dice on BraTS 2021 (MedSegDiff), and MedSegDiff-V2, which reports 0.901 mean Dice on AMOS against 0.878 for nnU-Net (MedSegDiff-V2). We found no reported result on Lacuna for any diffusion segmenter on lesion-free images, so the healthy-scan numbers will be new for every method.

Ablations isolate the ingredients one at a time. The first is a single network with the $w(t)$ timestep schedule and Dice-on-positives masking, but no experts and no stop-gradient; this tests whether timing alone is enough. The second is a dense residual head that receives Dice gradients behind a stop-gradient but has no gate; this tests isolation without routing. The third is hard timestep-band experts with a fixed assignment by noise level and no learned gate. The fourth is the full model with its stop-gradient removed.

All variants share the backbone, data splits, number of sampling steps, an equal number of trainable parameters (the dense head is sized to match the active expert parameters), and the same hyperparameter search budget, with eight configurations each, selected on in-distribution validation Dice only. Beating the field means two things. On healthy images, the full model must match or exceed cross-entropy-only training on false-positive rate. On lesion images, it must match or exceed the static hybrid on Dice and 95th-percentile Hausdorff distance (HD95), the boundary-error metric BraTS uses (BraTS 2021). It must do both at once, and with non-overlapping confidence intervals across three seeds against the transplanted Hard Fine-tuning.

Why not just…?

Why not just turn Dice off on images whose ground-truth mask is empty? This is the cheapest fix and it is a real threat; our full model includes it, and the first ablation tests it alone. It removes the degenerate denominator during training. It does not address the reported failure: models trained with Dice become overconfident and fire on healthy tissue they never saw labelled, as with ulcerative colitis frames in Endotect. Those errors come from shared features that Dice has sharpened, not from empty-mask batches. If this ablation matches the full model on healthy-scan false positives, we will report that gradient isolation is unnecessary and the contribution shrinks to the diagnosis and the compute savings.

Why not just use a timestep-dependent loss weight in a single network? The $w(t)$-only ablation is exactly this. In one network, the parameters that decide presence at high noise are the same ones that Dice updates at low noise, so the schedule cannot keep Dice away from the presence decision. That is our hypothesis, not established fact. The evidence for it would be a gap in healthy-scan false-positive rate between this ablation and the dense stop-gradient head. If no gap appears, timestep weighting is sufficient and the experts are not needed for accuracy.

Why not just add a dense second head instead of routed experts? We expect the dense stop-gradient head to capture most of the robustness benefit, and we say so plainly. The case for routing rests on cost. Diffusion segmentation already costs about 20 seconds per image over 100 steps versus 0.02 seconds for a ResUNet (MedSegDiff). FlexControl showed that routing at a 50% compute budget can even improve quality, with FID 14.80 versus 16.78 for the always-on ControlNet-Large (FlexControl). FID there measures image realism, not segmentation, so this suggests rather than proves a benefit for us. Routed experts must therefore match the dense head's accuracy at measurably lower compute per sample, or the dense head is the better recommendation.

Why not just use Hard Fine-tuning across epochs? It is the best documented fix, but it was validated only on single-pass CNNs. Applied to a diffusion model, its final Dice phase would update every parameter at every timestep, including the early-step presence pathway. The transplanted baseline tests directly whether this reintroduces hallucinations.

Proposed Experiments

We use the two benchmarks in the file that contain real negative images. These are DFUC 2022, with roughly 4,000 foot photographs including healthy remission cases, and Endotect 2021, with polyp frames plus ulcerative colitis frames that contain no polyps. Both come with the published CNN reference points above. For a radiology setting we use 2D slices from BraTS 2021. Lesion-free slices from the same patients serve as in-scanner negatives. This construction is ours, so we report it separately from native negatives. AMOS checks that the method does not hurt multi-organ segmentation, where small structures such as adrenal glands are hard and empty masks per organ occur naturally in cropped slices.

Metrics are Dice and HD95 on lesion images. On lesion-free images we report the image-level false-positive rate (any predicted component above 10 pixels), along with false-positive area. We also report expected calibration error, given the overconfidence attributed to Dice, and wall-clock time and FLOPs per sampled mask at 10 and 50 steps.

Two checks decide the outcome. The first is the trajectory diagnosis on the static-loss baseline. If false positives on healthy images do not originate at high noise, the timestep-gating rationale fails, and we redirect the work toward the masking and epoch-scheduling baselines. The second is a pair of comparisons on DFUC and Endotect: the full model against the $w(t)$-only ablation and against transplanted Hard Fine-tuning. The proposal succeeds if the full model lowers the healthy-image false-positive rate relative to both while keeping lesion-image Dice within 0.5 points of the static hybrid. Separately, the routed version must beat the dense head on compute at equal accuracy. We will also plot gate activation against timestep. If it shows the sparse-early, dense-late pattern FlexControl found, and gates near zero on healthy images, that is direct evidence the experts took the roles we designed.