Where Diffusion Attention Disagrees: Timestep Variance as a Boundary Band for Low-Contrast Segmentation

Text-to-image diffusion models are generators that learn to turn random noise into images step by step. Inside them, a cross-attention layer decides which image locations each word in the prompt affects. Read out for a word like "polyp," that layer gives a rough map of where the object is. Prior work treats this map as one static heatmap, averaged over the denoising steps and then cleaned up. We propose to look at how the map changes across noise levels. The central hypothesis is that attention from different timesteps agrees in the confident interior of an object and disagrees in the low-contrast rim, where the object blends into its surroundings. Timestep disagreement would then be a free, per-pixel estimate of boundary ambiguity.

The opportunity is in medical images such as colonoscopy frames. There a polyp's edge is a gradual change in mucosal color and texture, not a sharp step. The best discriminative segmenters handle this by splitting the object into a confident interior and an uncertain rim, then letting the rim borrow information from the interior. Today they draw that rim with fixed-width erosion and dilation of a coarse mask. We would replace the fixed rim with the disagreement band from diffusion attention. We would also use the text-conditioned attention as a second initial localizer, which can catch camouflaged lesions that the coarse discriminative mask misses entirely.

Core Claim

The per-pixel variance of a prompt token's cross-attention, computed over several noise levels of the same clean image, marks boundary error better than two existing signals: morphological rim extraction and the entropy of a discriminative coarse prediction. Using this variance as a soft boundary band, the rim of the interior-to-boundary refinement, should improve boundary metrics most on unseen, small, concealed polyps. The claim is falsifiable before any architecture is built. If attention variance does not predict where the ground-truth boundary lies, and where a baseline segmenter errs, better than coarse-mask entropy does, the proposal is dead.

Relation to Prior Work

The attention-localization line supplies the readout. Three papers establish the tools:

  • DiffuMask aggregates cross-attention across resolutions and timesteps into pixel annotations.
  • From Text to Mask sharpens cross-attention by spreading it through self-attention affinities (its SelfCross module) and then a Dense CRF, reaching 76.1% mIoU on Pascal VOC.
  • Generate to Ground shows that medical-text-conditioned diffusion attention nearly doubles a discriminative model on MS-CXR grounding (0.54 vs 0.266 mIoU).

All three collapse the timestep axis by averaging. They discard exactly the signal we want to study. Generate to Ground also reports failure on irregular, low-contrast findings such as pneumothorax, and it is scored with boxes, not boundaries. That motivates our focus on pixel-level edges.

The boundary line supplies the refinement mechanism and constrains the design:

  • PraNet erases the confident foreground so later layers attend to the rim ("reverse attention"). It reaches 0.898 mDice on Kvasir but falls to 0.628 on unseen ETIS.
  • MISNet adds axial boundary attention.
  • Polyper lets rim features query interior features through cross-attention between image regions. It defines the rim by fixed-kernel erosion and dilation, and names this rigidity as a limitation.

All three depend on a coarse first mask. If that mask misses a lesion, nothing downstream can recover it. We keep Polyper's rim-queries-interior refinement. We change only where the rim comes from, and we add an independent text-driven localization path.

Two diffusion-for-medicine papers set the boundaries of the idea:

  • DiffuGTS is the closest existing work. It uses diffusion attention maps only as a rough region, then refines boundaries by inpainting a pseudo-healthy image over many steps and taking residuals. It gains about three Dice points on unseen liver tumors, at acknowledged latency cost. Our refinement is feed-forward and never synthesizes tissue, so it cannot hallucinate healthy texture across an indistinct edge.
  • SDSeg argues that concatenating image latents beats cross-attention for dense medical segmentation. It uses a single reverse step to stay fast. This is a direct challenge. We answer it by using cross-attention only as an uncertainty and localization signal, alongside a dense decoder, and by testing SDSeg-style concatenation as a baseline.

Research Program

We use a frozen Stable Diffusion U-Net. The only trainable part on the generative side is a small learned embedding for a "polyp" token, trained on the segmentation training split. Everything else trainable sits in a lightweight decoder.

For each image, we encode it to a latent and noise that latent to a small set of timesteps $\mathcal{T}$ (four, spread across the schedule). We run all four noised copies through the U-Net in one batched forward pass. There is no iterative sampling, so cost is close to SDSeg's single step times four.

Let $a_t(i)$ be the polyp token's cross-attention at pixel $i$ for timestep $t$. We aggregate $a_t(i)$ across layers and upsample it to full resolution. We then normalize each map to unit maximum. From these maps we compute two quantities:

$$\mu(i)=\frac{1}{|\mathcal{T}|}\sum_{t\in\mathcal{T}}a_t(i), \qquad \sigma^2(i)=\frac{1}{|\mathcal{T}|}\sum_{t\in\mathcal{T}}\big(a_t(i)-\mu(i)\big)^2 .$$

These define two soft region weights. The interior weight $w_{\text{int}}(i)$ is high where $\mu$ is high and $\sigma$ is low. The band weight $w_{\text{band}}(i)=\sigma(i)/\max_j \sigma(j)$ is high where timesteps disagree. Neither uses a morphological kernel. We apply self-attention propagation in the style of SelfCross to $\mu$ before computing these weights, to reduce the known bleeding of raw attention.

The decoder takes a discriminative backbone's features concatenated with the U-Net features, following the SDSeg style. It produces a coarse mask that is fused with $\mu$, so a lesion found by either path survives. The decoder then runs Polyper's boundary-sensitive attention with $w_{\text{band}}$ as soft query weights and $w_{\text{int}}$ as soft key/value weights. Training uses Dice plus cross-entropy plus a contour term on the final mask, following Niu et al., who report that a contour branch cut 95% Hausdorff distance on Spleen CT from 7.5744 to 5.8814.

The analysis phase comes first and gates everything else. On held-out images we measure how well $\sigma$ ranks two kinds of pixels:

  • pixels within a few pixels of the ground-truth boundary;
  • pixels a PraNet model gets wrong.

We report area under the ROC curve for both. We compare $\sigma$ against three alternatives:

  • a Polyper-style dilation-minus-erosion rim;
  • the per-pixel entropy of PraNet's own coarse output;
  • variance across noise seeds at a single fixed timestep.

The last comparison tells us whether the signal comes from the multi-scale nature of timesteps or merely from stochasticity.

Baselines

The naive reference point has two parts. The first is the raw averaged cross-attention map for the learned polyp token, thresholded. The second is the same map after SelfCross propagation and Dense CRF, as in From Text to Mask. This tells us how much the variance-guided decoder adds over a post-processed attention readout. It gets the same learned token and the same training split.

The strongest discriminative competitors are PraNet, MISNet, and Polyper. PraNet reports 0.898 mDice on Kvasir and 0.628 on ETIS. MISNet reports 0.903 on Kvasir and 0.918 on CVC-ClinicDB. Polyper, the most direct competitor because it uses the same rim-queries-interior mechanism, reports 86.51% mDice on ETIS (versus SegT's 81.00%). That ETIS figure is far above PraNet's, which suggests a different training protocol. We will therefore retrain all three under one protocol rather than compare against published numbers. The protocol is:

  • train on the standard Kvasir-plus-ClinicDB split;
  • test on Kvasir, ClinicDB, CVC-ColonDB, CVC-300, and ETIS;
  • use the same augmentation, input resolution, and an equal hyperparameter search budget.

Beating Polyper means a higher mean ETIS and ColonDB Dice across three seeds, with non-overlapping confidence intervals, plus a lower boundary distance.

The strongest diffusion competitor is SDSeg, retrained under the same protocol. Its feed-forward cost is closest to ours, and it embodies the concatenation-over-cross-attention argument. For SDSeg on Kvasir-SEG and CVC-ClinicDB, no reported result was found on Lacuna, so our rerun is the reference.

The key ablation holds the whole architecture fixed and swaps only the source of the band. The four sources are:

  • Polyper's fixed morphological rim;
  • the coarse-mask entropy;
  • the attention mean alone, with a rim derived from it;
  • seed variance at a fixed timestep.

If timestep variance does not beat entropy in this ablation, the diffusion component is not earning its cost.

Why not just…?

Why not just give Polyper an adaptive rim from its own prediction entropy? This is the most serious threat. It needs no diffusion model and directly fixes Polyper's rigid-kernel limitation. Our band-source ablation tests exactly this swap inside an otherwise identical model. We would believe our claim only if timestep variance wins both there and in the pre-build calibration study. Entropy has a structural weakness here: a confident but wrong coarse mask has low entropy precisely where it misses a camouflaged lesion. We therefore expect the gap to appear mainly on ETIS small polyps. If it does not, we will report that entropy suffices and drop the generative band.

Why not just post-process diffusion attention with SelfCross and a CRF? From Text to Mask already shows this pipeline is strong on natural images. But a CRF relies on color contrast, which is exactly what is missing at a polyp margin. Polyper's ablation also shows that dropping explicit region separation costs 4.40 mIoU points. The naive-baseline comparison measures this gap directly on polyps.

Why not just use SAM or CLIP-style features? The file reports that discriminative vision-language models lose spatial detail on medical images. Generate to Ground shows generative attention at 0.54 versus BioViL's 0.266 mIoU on MS-CXR. DiffuGTS reports large zero-shot gains over such methods. No polyp-specific SAM number appears in the file, so we do not claim one. What matters is that SAM supplies no timestep axis, and the proposal's contribution is that axis.

Why not just fine-tune SDSeg, since concatenation reportedly beats cross-attention? We include SDSeg as a baseline and also reuse its concatenation path in our decoder. The question is not "cross-attention or concatenation" but whether cross-attention variance adds something concatenation lacks. The fused model beating SDSeg while SDSeg-plus-entropy-band does not would answer SDSeg's objection.

Proposed Experiments

Calibration study (gate). We compute the ROC area of $\sigma$, entropy, morphological rims, and seed variance for boundary pixels and for baseline-error pixels. We report it separately for large and small polyps and for ETIS. The rest of the program proceeds only if timestep variance leads.

Main segmentation comparison. The metrics are mDice, mIoU, 95% Hausdorff distance, and a boundary F-score at a few pixel tolerances. We also report error restricted to a ground-truth boundary strip, since whole-mask Dice hides rim quality.

Recovery of missed lesions. On ETIS, we count polyps that PraNet's and Polyper's coarse masks miss entirely, then measure how many the fused text-attention path detects. This tests the mislocalization failure named in the boundary literature.

Cost. We report latency per frame against PraNet and SDSeg. We also vary $|\mathcal{T}|$ from two to eight to find the smallest set that keeps the calibration advantage.

Secondary transfer check. We run the calibration study, without training, on MS-CXR using a CXR-BERT-conditioned model in the style of Generate to Ground. The question is whether variance also concentrates at the box edges of findings like pneumothorax, the case that paper reports failing.