Many hospitals want to generate one scan type from another, for example a synthetic CT from an MRI so a patient can be planned for radiotherapy without a second scan. A cheap way to do this in 3D is to run a 2D diffusion model (a generator that learns to turn random noise into images by removing noise step by step) on each slice of the volume. The catch is that slices generated independently do not line up. Vessels break, organ edges form staircases, and the volume flickers along the depth axis. Current fixes couple neighbouring slices, either by sharing noise or by predicting slices jointly, and they apply that coupling with the same strength at every denoising step. Separately, work on natural images shows that diffusion models fix global layout in the first, noisiest steps and spend the later steps on fine texture.
This note proposes to join those two observations. Slices should be tightly coupled early, when 3D geometry is being decided, and progressively released late, when each slice is filling in in-plane detail. The test is whether this changes anything that matters clinically: radiation dose computed on the synthetic CT, and tumor segmentation on a synthesized MRI sequence. Pixel similarity, which the field already knows can mislead, is not the target. A second question makes the idea useful beyond one dataset. Can the release point be predicted from a label-free measurement of when through-plane structure settles, so that a schedule tuned on one modality pair transfers to another?
Core Claim
For 2D diffusion models used slice-by-slice on 3D cross-modality translation, the right amount of coupling between slices depends on the timestep. A schedule that couples strongly early and weakly late should beat the best constant coupling strength on clinical endpoints, especially thin or focal structures and dose. We further claim the schedule's switch point tracks a measurable property of each model: the step at which low through-plane frequencies stop changing. That property lets the schedule be set without clinical labels on a new modality pair. If a single well-tuned constant coupling matches the schedule on clinical endpoints, the claim is false, and the experiments are built so this can be observed.
Grounding in Prior Work
Three papers motivate the timing half of the idea. Yi et al. used Fourier analysis to show that denoising rebuilds low frequencies before high ones, and dropped text conditioning after the first ~20 of 50 steps with a "25%+ reduction in inference latency" and preserved FID. Chefer et al. (Attend-and-Excite) found that objects not attended to within the early steps are permanently lost. Hertz et al. (Prompt-to-Prompt) froze layout by injecting attention maps only in early steps.
One further paper warns against steering at every step. Dinh et al. (Compress Guidance) showed that guidance applied uniformly makes samples overfit the guiding model: accuracy was 90.8% on the guiding classifier but 34.2% on a held-out one. All four works are 2D and natural-image. None schedules anything along the depth axis.
Four papers supply the volumetric machinery, and all couple slices uniformly in time. ISCS (Du et al.) traces slice flicker to independent per-slice noise. It fixes this without retraining by drawing boundary noise vectors and interpolating intermediate slices on the hypersphere, which kept standard Gaussian statistics and gained "nearly 2 dB over the baseline DDS method" on AAPM sparse-view CT. It is an inverse-problem method, not a translation method, and it uses the same interpolation at every step. The official code is available at ISCS codebase. Make-A-Volume inserts 1D depth-axis layers into a 2D latent diffusion model and forces a shared timestep across slices. Choo et al. co-predicts a shared trajectory across adjacent slices. cWDM avoids the problem with full-volume 3D diffusion in a wavelet space. We reuse ISCS's noise construction as our coupling primitive and change only when it acts.
Three results constrain the design. First, SynthRAD2023 found that 2D and 2.5D models beat 3D ones for MRI-to-CT. This justifies keeping a 2D backbone. The same study found that 3D patch models won for CBCT-to-CT, so any 2D approach must be checked on that task too. Second, SynthRAD2023 found that high SSIM "does not necessarily correlate with accurate dose distributions". Third, Konz et al. showed that FID rewards style over anatomy. Together these rule out PSNR, SSIM or FID as the primary metric.
SegGuidedDiff supplies the evaluation idea we borrow: judge synthetic images by how a downstream segmentation model performs on them. We extend that 2D protocol to 3D translation.
One claim appears to cut against us. ISCS reports that it removes flicker "without blurring small focal lesions" on DeepLesion, which suggests uniform coupling may already be harmless. We take this seriously. Our hypothesis is not that uniform coupling blurs lesions visibly. It is that correlated late-step noise imposes shared fine detail on slices whose true fine detail differs, especially at coarse slice spacing. That effect would show up in dose and in the number of focal lesion components more than in PSNR, and ISCS did not measure either.
Research Program
The mechanism is deliberately minimal. Let $\epsilon^{\text{c}}_{t,z}$ be the ISCS-interpolated noise for slice $z$ at step $t$, and $\epsilon^{\text{i}}_{t,z}$ an independent standard Gaussian draw. We inject
$$\epsilon_{t,z} = \sqrt{w(t)}\,\epsilon^{\text{c}}_{t,z} + \sqrt{1-w(t)}\,\epsilon^{\text{i}}_{t,z},$$
where $w(t)\in[0,1]$ is the coupling weight at step $t$. Because the two terms are independent and each is marginally $\mathcal{N}(0,I)$, the mixture stays standard Gaussian for every $w$, so the sampler's statistics are untouched. Only the correlation between adjacent slices changes, and it scales linearly with $w(t)$. The schedule family is a sigmoid step from $w=1$ to a floor $w_{\min}$ at switch point $t^\star$ with width $\delta$. The constant schedule $w(t)=c$ is the special case that is flat in $t$.
For models with deterministic cross-slice modules, we apply the same schedule as a gate on the module's residual. These are a Make-A-Volume-style depth layer or an ISTA-style joint correction. The gate is inference-only and needs no retraining.
For the label-free calibration, we run the sampler on unlabelled source volumes. At each step we measure the depth-axis power spectrum of the model's predicted clean volume. We record $t_{\text{settle}}$, the step at which the energy in the lowest through-plane frequency band changes by less than a small fraction per step. This extends Yi et al.'s in-plane analysis to the depth axis. The prediction is that the clinically best $t^\star$, found by search on a labelled pair, lies near $t_{\text{settle}}$. If it does, setting $t^\star := t_{\text{settle}}$ on a new modality pair should recover most of the searched schedule's benefit.
Backbones are 2D conditional diffusion models, one per task, trained once. All coupling variants are inference-time, so every comparison shares weights.
Baselines
The naive reference is independent per-slice sampling with the same 2D backbone ($w\equiv 0$). This is the Palette-style setting that Make-A-Volume reports as badly degraded, with MAE 26.806 against its own 4.801 on SWI-to-MRA. It fixes the floor that every coupling method must clear.
The strongest direct competitor is uniform ISCS ($w\equiv 1$) from Du et al., run from the released code on the identical backbone and step count. It is training-free, preserves Gaussianity, and targets exactly the failure we address. No reported result was found on Lacuna for ISCS on MRI-to-CT or BraTS translation, so we generate these numbers ourselves.
The ablation that isolates our ingredient is the best constant coupling $w\equiv c$. We search $c$ over the same grid and the same validation budget as the schedule's parameters $(t^\star,w_{\min},\delta)$, with an equal number of evaluated configurations. Beating ISCS but not this tuned constant would mean the gain is a strength effect, not a timing effect. We also run the reversed schedule (weak early, strong late); timing theory predicts it is worst.
Among deterministic coupling methods, we run the Make-A-Volume depth-layer model both ungated and with our gate. On BraTS 2024, the architectural competitor is native 3D cWDM, which reports PSNR/SSIM of 29.74/0.956 (T1), 27.31/0.936 (T1ce), 28.86/0.952 (T2) and 27.83/0.935 (FLAIR). On SynthRAD2023 MRI-to-CT, the reference is the challenge's top teams, reported by Huijben et al. at SSIM $\ge 0.87$ and gamma pass rates of photon $\ge 98.1\%$ and proton $\ge 97.3\%$.
"Beating" means a statistically significant paired improvement over the tuned constant on the primary clinical endpoint of each task, with PSNR no worse than that of uniform ISCS. Paired significance is assessed with a patient-level bootstrap. The primary endpoints are dose error and proton gamma pass rate for SynthRAD, and tumor-subregion Dice for BraTS. We do not claim to beat challenge winners with a different backbone. The challenge numbers mark where the backbone sits, not the test of the idea.
Why not just…?
Why not just tune a single constant coupling strength? This is the real threat. A moderate constant correlation might already balance flicker against shared late detail. It is the central ablation above, given an equal search budget. If it ties the schedule on dose and lesion metrics across all three tasks, we drop the timing claim and report that coupling strength, not timing, is what matters.
Why not just smooth the stacked 2D output along the depth axis? Post-hoc filtering removes flicker but cannot restore topology that diverged early, such as a vessel generated on the wrong side on slice $z+1$. Gu et al. report that bad layouts seeded early cannot be repaired late without distortion. We include a depth-axis Gaussian filter, tuned on validation, as an extra baseline. We expect it to match on flicker scores but lose on proton gamma and on small-lesion component counts, where blurring is harmful.
Why not just use a native 3D model? For MRI-to-CT, SynthRAD2023 found that 2D and 2.5D models beat 3D ones, and native 3D attention at clinical resolution is memory-bound. For CBCT-to-CT, however, 3D patch models won. We therefore include CBCT-to-CT as the task where our approach is most likely to lose, and cWDM on BraTS as a direct 3D comparison. If native 3D dominates on clinical endpoints at equal inference compute, the contribution narrows to a cheaper alternative, and we will say so.
Proposed Experiments
The first check is the timestep-contribution map. Couple slices only within one window of steps at a time and measure how the clinical endpoints change. The theory predicts that early windows carry most of the benefit to topology and dose, and that late windows are neutral or harmful. A flat map falsifies the idea before any schedule search.
The second check is the main comparison on SynthRAD2023. It covers MRI-to-CT and CBCT-to-CT, using the challenge's dose-based endpoints (MAE of dose, DVH parameters, and photon and proton gamma) alongside MAE and SSIM. We use the challenge's own evaluation protocol if it can be obtained; otherwise we report that the dose engine is our own.
The third check is on BraTS 2024 missing-sequence synthesis. A tumor segmentation model trained on real complete studies is applied with the synthesized sequence substituted in. We report subregion Dice, tumor volume error, and connected-component counts of enhancing tumor, which capture the splitting and duplication failure noted for pseudo-3D methods.
On every task we also report a through-plane consistency score: depth-axis gradient spectra and flicker. This confirms the schedule does not reintroduce staircases late in sampling.
The fourth check is transfer across modality pairs. Search $t^\star$ on MRI-to-CT, then compare three settings on T1-to-FLAIR (BraTS) and CBCT-to-CT: the transferred $t^\star$, the label-free $t_{\text{settle}}$, and the $t^\star$ searched on each target pair. Success means $t_{\text{settle}}$ recovers most of the searched schedule's gain over the tuned constant on each target pair.
If time allows, we add SWI-to-MRA with vessel-graph connectivity as a topology endpoint.
The decisive outcome is simple. The schedule must beat the budget-matched constant on at least the dose and lesion endpoints, and the label-free calibration must transfer. Either failure is publishable, as a clean negative result on timestep structure in volumetric coupling.
References
- Towards Understanding the Working Mechanism of Text-to-Image Diffusion Model
- Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models
- Prompt-to-Prompt Image Editing with Cross-Attention Control
- Compress Guidance in Conditional Diffusion Sampling
- Improving 2D Diffusion Models for 3D Medical Imaging with Inter‑Slice Consistent Stochasticity
- Make-A-Volume: Leveraging Latent Diffusion Models for Cross-Modality 3D Brain MRI Synthesis
- Slice-Consistent 3D Volumetric Brain CT-to-MRI Translation with 2D Brownian Bridge Diffusion Model
- cWDM: Conditional Wavelet Diffusion Models for Cross-Modality 3D Medical Image Synthesis
- Generating synthetic computed tomography for radiotherapy: SynthRAD2023 challenge report
- Rethinking Perceptual Metrics for Medical Image Translation
- Anatomically-Controllable Medical Image Generation with Segmentation-Guided Diffusion Models
- Mitigating Noise-Induced Layout Priors for Object Counting in Diffusion Models