Factorized Critics for Few-Step 3D Medical Diffusion

Diffusion models are generators that turn random noise into an image by removing a little noise at a time. They produce excellent 3D medical images, but they usually need hundreds to a thousand denoising passes per scan. The standard way to cut this to one or two passes is adversarial step distillation: a second network, called a critic or discriminator, judges whether each large denoising jump lands on a realistic image. This keeps outputs sharp, where a plain averaging loss would blur them. In 3D brain MRI, this approach has hit a contradiction. fastWDM3D carried a 2D adversarial diffusion model into 3D for healthy-tissue inpainting, found that the volumetric discriminator hurt quality, and removed it. McCaD found an adversarial critic essential for 4-step MRI synthesis, but it avoided 3D entirely by working on 2D slices.

We propose to settle this contradiction with one architectural change. The critic is built from factorized 3D convolutions: each cubic $3\times3\times3$ filter is split into an in-plane $1\times3\times3$ filter and a through-slice $3\times1\times1$ filter, with a nonlinearity between them. The two-sentence version is this. 3D adversarial distillation failed in MRI because a full 3D critic trained on about a thousand scans overfits and stops giving useful gradients. A factorized critic shares its in-plane filters across every slice and models cross-slice continuity with a small separate branch, so it should stay informative long enough to make 1–2 step volumetric generation sharper than the discriminator-free fastWDM3D model, without the slice-to-slice seams a 2D critic allows.

Core Claim

Our main claim is about why 3D critics fail, not only that a new critic works. We hypothesize that the failure reported by fastWDM3D comes from overfitting in the critic. A monolithic $3\times3\times3$ critic has 27 weights per channel pair. It sees roughly 1,200 training volumes in the BraTS inpainting challenge. It quickly memorizes the real scans, its scores for real and generated scans separate, and the gradient it passes back to the generator becomes uninformative or erratic.

A factorized critic changes this in three ways. It uses 12 weights per channel pair, the parameter saving documented in Fast Spatio-Temporal Residual Network for Video Super-Resolution. Its in-plane filters see every slice of every scan as a training example. It also gains the extra nonlinearity and easier optimization that A Closer Look at Spatiotemporal Convolutions for Action Recognition reports for R(2+1)D over full 3D networks. That paper is evidence from video classification, not from adversarial training, so it motivates the design but does not establish the transfer. The transfer is what we test.

The claim makes three falsifiable predictions. First, with the generator held fixed, a factorized critic shows a smaller gap between its scores on training and held-out real scans than a monolithic 3D critic of matched parameter count. Second, this smaller gap goes with better inpainting at 1 and 2 steps. Third, the through-slice branch matters specifically for coherence across slices, which tri-planar 2D critics cannot enforce.

Research Program

The generator is a 3D wavelet diffusion model in the style of fastWDM3D. It uses a variance-preserving noise schedule and the same full-image and masked-region reconstruction losses. The only change is an adversarial term at each large jump, in the style of denoising-diffusion GANs (Latent Denoising Diffusion GAN; Özbey et al.). The generator objective is

$$\mathcal{L}_G = \mathcal{L}_{\text{rec}} + \lambda\,\mathbb{E}_t\big[-\log D_\phi(\hat{x}_{t'}, x_t, t)\big],$$

where each term means the following. $\mathcal{L}_{\text{rec}}$ is fastWDM3D's reconstruction loss. $x_t$ is the noisy volume at step $t$. $\hat{x}_{t'}$ is the generator's prediction after one large jump to an earlier step $t'$. $D_\phi$ is a time-conditioned critic. $\lambda$ weights the adversarial term. Setting $\lambda=0$ recovers fastWDM3D exactly, which makes the "do nothing" comparison clean.

The critic $D_\phi$ is a residual network of factorized blocks: an in-plane $1\times3\times3$ convolution, a normalization layer and nonlinearity, then a through-slice $3\times1\times1$ convolution. Two scalar heads read the shared features. One head pools over slices after the in-plane stages and judges texture realism. The other reads the through-slice features and judges whether tissue boundaries continue smoothly across slices. Both heads see the tumor mask as an input channel, so judgment concentrates on the inpainted region.

This mirrors a lesson from video. AAD-1 found that frame-wise critics let a one-step video generator collapse to static output, and fixed this with an expensive holistic critic. Our through-slice branch is the cheap volumetric counterpart of that holistic view. Echocardiography work that separates tissue appearance from wall motion with (2+1)D networks (Hughes et al.) suggests the split suits medical volumes. The program has three parts: building this critic, measuring critic overfitting directly during training, and running ablations that separate the factorization itself from the parameter savings it brings.

Baselines

The first reference point is doing nothing: fastWDM3D itself, with no critic. It reports SSIM 0.8571, MSE 0.0079 and PSNR 22.26 dB on the BraTS inpainting test set, using 2 steps and 1.81 seconds per scan. It is also the strongest published fast method on this task in the file. That test set is not public, so we will retrain fastWDM3D under its published settings on our own held-out split of the 1,200 training scans. We will use the same steps, data and compute as our method, and report both our reproduction and the cited numbers. Beating it means a statistically significant improvement in masked-region SSIM and PSNR on the same split at 2 steps, plus a larger margin at 1 step. If challenge submission is possible, we will also submit to the official test set.

The second baseline is the design fastWDM3D abandoned: the same generator with a monolithic $3\times3\times3$ critic in the style of WDDGAN. No reported result was found on Lacuna for a 3D critic's inpainting scores, so we rerun it ourselves. We will build two versions. One has the same width as our critic. The other is parameter-matched by narrowing its channels. The parameter-matched version is the crucial control: if it does as well as the factorized critic, the gain came from capacity, not factorization.

The third baseline is the strongest simple alternative: three 2D critics applied to axial, coronal and sagittal slices, with outputs summed. It shares the slice-based stability of McCaD, which reports 28.87 dB PSNR and a downstream tumor-segmentation Dice of 0.7602 in its own 2D synthesis setting. Those numbers come from a different task and are not directly comparable to ours.

Two ablations isolate the key ingredient. One removes the nonlinearity between the in-plane and through-slice filters. The other removes the through-slice head. Every critic variant gets the same tuning budget of 12 runs over $\lambda$, learning rate and R1 weight, and is evaluated over three seeds.

Why not just…?

Why not just keep fastWDM3D and skip the critic? This is a real threat, because inpainting is heavily constrained by the healthy tissue surrounding the mask. There may be little ambiguity left for a critic to resolve. We address this by sweeping mask size and step count. At small masks with 2 steps, we expect a tie. At 1 step and with large masks, the remaining uncertainty is larger and plain regression should blur. If the critic gives no gain even there, we will report that the discriminator is unnecessary for this task and drop the claim.

Why not just regularize the monolithic 3D critic better? fastWDM3D may simply not have tuned it well. The parameter-matched monolithic critic, given the same R1 and tuning budget, answers this directly. Our overfitting diagnostic shows whether any remaining gap tracks memorization, which is what our mechanism predicts.

Why not just use three orthogonal 2D critics? They are cheap and stable. Each plane, though, sees only within-slice structure in its own orientation, so no single critic sees continuity along all three axes jointly. By analogy with the frame-wise collapse in AAD-1, we expect seams that each plane individually accepts. If tri-planar critics match ours on the cross-slice coherence metric, the through-slice branch adds nothing and we will say so.

Why not just use a perceptual or feature-matching loss? These losses still average over plausible outputs at large jumps, which is the blurring that adversarial distillation exists to avoid (Kang et al.). Even so, we include a feature-matching variant built from the critic's own frozen features, to check whether the adversarial game itself is needed.

Proposed Experiments

All experiments use the BraTS 2023 healthy-tissue inpainting data. We split the 1,200 training scans into train and held-out sets and also use the 51 validation scans. We report masked-region SSIM, PSNR and MSE following fastWDM3D, at 1, 2 and 4 steps, along with wall-clock time per scan.

Cross-slice coherence is measured by computing PSNR and gradient statistics on sagittal and coronal reformats of axially-dominant errors. This is where tri-planar critics should fail. Critic overfitting is tracked throughout training as the gap in critic scores between training and held-out real volumes.

Three outcomes would decide the claim. The factorized critic must beat our fastWDM3D reproduction at 1 step. It must beat the parameter-matched monolithic critic, and that advantage must line up with a smaller overfitting gap. It must also beat the tri-planar critics on the reformatted-slice coherence metric.

As a secondary check, we test whether the result transfers to a harder setting: 3D multi-contrast synthesis on the BraTS contrasts, with downstream segmentation Dice scored in the manner of McCaD.