Tracking Healthy Anatomy Through Depth for Diffusion-Based Anomaly Detection

One way to find a tumor or bleed in a brain scan without ever labeling one is to train a generative model only on healthy scans. Given a patient scan, the model "heals" it into its best guess of a healthy version, and wherever the patient and the healed copy disagree is flagged as abnormal. The best current versions use diffusion models, which are generators that learn by gradually denoising random noise into realistic images. They heal each 2D slice of a 3D scan independently, because true 3D diffusion is too memory-hungry. The result is healed volumes that tear from one slice to the next, and anomaly maps that flicker along the depth axis. The usual fix is to go fully 3D. That forces much heavier compression of each image, and heavy compression erases exactly the small lesions one most wants to catch.

We propose a middle path borrowed from video tracking. Treat the depth axis like time. Heal slice $z$ with a 2D diffusion model that is shown the already-healed slice $z-1$, after that neighbor has been warped into slice $z$'s geometry by a deliberately low-capacity deformation field. The warp carries healthy anatomy forward through depth, the way a tracker carries a deforming object across frames. Because the warp is too smooth to draw a lesion, and the thing being warped has already been healed, the abnormality has no path to leak into the reconstruction. Staying in 2D lets us spend the memory budget on in-plane resolution rather than on the depth axis. The bet is that this recovers most of the cross-slice coherence of 3D models while keeping the small-lesion sensitivity of 2D ones.

Core Claim

The claim has two parts, and both can be tested. First, at a fixed GPU memory budget, depth-propagated 2D healing will beat a native 3D latent diffusion model on lesions smaller than a few latent cells. It will also match or beat that model on cross-slice coherence, because 3D models must compress more aggressively to fit.

Second, the capacity of the deformation field sets a trade-off that previous work has only stated informally. The research file calls it the "deformable shortcut" tension: flexible alignment modules are needed to follow anatomy, but a flexible enough module can also reproduce the anomaly and zero out the residual. We predict a U-shaped curve in the spacing of the deformation control grid. If the grid is too fine, the warp can represent lesion-shaped displacements, and detection collapses. If it is too coarse, anatomy misaligns and false positives appear along tissue boundaries. A band in between should be clearly better than both ends. If no such band exists, the proposal fails.

Relation to Prior Work

The method starts from Fast Unsupervised Brain Anomaly Detection and Segmentation with Diffusion Models. That work compresses brain slices with a vector-quantized autoencoder, trains a diffusion model on healthy latents, and localizes anomalies in the middle of the reverse chain, which it finds to be a "sweet spot" between $t=400$ and $t=600$. It works slice by slice and names native 3D as unsolved. We keep its partial-noising recipe and its sweet-spot finding. Unlike it, we couple slices through depth, and we give up some latent compression.

Related "anomalies as noise" diffusion approaches such as Bercea et al. share the same per-image scope. The failure we target is documented by Deep convolutional networks for pancreas segmentation in CT imaging and Conditional PGGAN-based Data Augmentation for Brain Metastases Detection. Both show that independent 2D predictions or generations produce fragmented, misaligned 3D shapes. These papers motivate the work; they do not supply the method.

The resolution constraint comes from two papers. S4ND shows that pulmonary nodules of 3–30 mm are washed out by coarse multi-scale grids in 3D detectors. LIP shows that fixed pooling dilutes small targets; learned pooling raised small-object AP on COCO from 21.9% to 24.0%. Both are supervised. We use them as evidence that compression stride governs small-target survival, and we test that directly in the unsupervised healing setting.

The propagation mechanism draws on deformable tracking. Heisele et al. initialize frame $t+1$ from the converged state of frame $t$. Joint tracking-and-segmentation work such as Wen et al. propagates masks with flow. Nieuwenhuis et al. bound deformation with shape and proportion priors so the tracker cannot bleed into the background. Our low-capacity warp is such a prior, repurposed: in our setting, what it must be unable to follow is the lesion.

Two nearer bridges use learned deformation between views or frames. GeoMAD predicts free sampling offsets between camera views of rigid parts inside a distillation detector. SCOTCH and SODA models shadow deformation trajectories in supervised video. Neither heals anything, and neither restricts its offsets. We deliberately restrict ours.

DeScarGAN is the closest precedent for reasoning about capacity. It removes the top U-Net skip connection so the generator can reshape anatomy without copying the input. We move that capacity decision out of the network wiring and into an explicit knob, the warp grid spacing, and we sweep it.

Research Program

The system has three components.

The first is a 2D denoising diffusion model trained on healthy slices. It works either in pixel space or in a latent space with light compression, meaning a small downsampling stride, chosen so it fits the same memory budget as the 3D baseline.

The second is a small registration network. Given two adjacent input slices, it outputs a B-spline displacement field defined on a control grid with spacing $s$ pixels.

The third is the conditioning pathway. The warped neighbor is concatenated to the noisy current slice at every denoising step.

Training uses healthy volumes only. The neighbor is the true slice $z-1$, and the registration network is trained with an image-similarity loss plus a smoothness penalty. At test time, slice $z$ is partially noised to the sweet-spot level $t^\*$ and denoised under the condition

$$ c_z = \mathcal{W}_{s}\big(\hat{x}_{z-1};\ \phi_s(x_{z-1}, x_z)\big), $$

where $x_z$ is the input slice, $\hat{x}_{z-1}$ is the healed previous slice, $\phi_s$ is the displacement field on grid spacing $s$, and $\mathcal{W}_s$ is warping by that field.

We run the chain top-down and bottom-up. The anomaly score at each voxel combines two terms:

$$ a_z = \tfrac{1}{2}\left(|x_z - \hat{x}^{\downarrow}_z| + |x_z - \hat{x}^{\uparrow}_z|\right) + \lambda\,|\hat{x}^{\downarrow}_z - \hat{x}^{\uparrow}_z|. $$

Here the arrows mark the two directions, and $\lambda$ weights disagreement between them. Disagreement is itself a signal: where the two chains heal differently, the model is unsure what normal anatomy looks like there.

The analysis has three threads. First, we measure lesion recall stratified by lesion diameter expressed in units of the effective compression stride. This directly tests the resolution-degradation hypothesis. Second, we sweep $s$ to trace the leakage/misalignment curve. Third, we inject synthetic lesions of known size and shape into healthy volumes and check whether the residual inside them collapses. This measures leakage directly. We also measure how far a lesion's first slice contaminates the healed slices after it.

Baselines

The naive reference is the slice-independent latent diffusion healer of Fast Unsupervised Brain Anomaly Detection and Segmentation with Diffusion Models, rerun with the same healthy training data and the same sweet-spot noising. It isolates what depth coupling adds. The file reports a synthetic MedNIST Dice of 0.920 for this model, against 0.895 for a transformer ensemble. For BRATS, WMH and MSLUB, the file only says the model is "competitive with state-of-the-art ensembles": no reported result found on Lacuna for volumetric Dice on these sets. We will therefore rerun it rather than quote it.

The strongest competitor is a native 3D latent diffusion healer. It is the obvious answer to the gap the source paper names. We give it the same memory budget and the same number of tuning trials, and we let it choose its own compression stride to fit. Beating it means two things: a higher lesion-wise recall on lesions below four stride units, and cross-slice coherence and overall Dice on healthy and diseased volumes within its confidence interval or better.

The key-ingredient ablations change one thing each. The first conditions on the healed neighbor without warping, which tests whether deformation matters. The second conditions on the raw, unhealed input neighbors, the 2.5D recipe; this tests whether healing before propagation prevents leakage. The third sweeps the grid spacing $s$ from a single pixel up to a near-rigid transform, which tests the capacity claim.

For context, we also report a slice-wise PatchCore-style feature detector. Benchmarking Unsupervised Anomaly Detection and Localization shows such detectors dominate 2D industrial inspection, at up to 99.4% image AUROC on MVTec AD. No brain-MRI number for it appears in the file.

Why not just…?

Why not just use 2.5D conditioning on the raw neighboring slices? This is the cheapest option and a real threat. Our worry is specific: lesions span several slices. A model conditioned on raw neighbors can copy the lesion from slice $z-1$ into its reconstruction of slice $z$, which is the copying failure DeScarGAN works around by removing a skip connection. The raw-neighbor ablation measures this directly as residual collapse inside injected lesions, as a function of how many slices each lesion spans. If raw conditioning does not collapse there, the healing-first design is unnecessary, and we will say so.

Why not just smooth the 2D anomaly maps along depth? A 3D Gaussian or connected-component filter on the per-slice residuals would remove much of the flicker for almost no cost. It could win on coherence metrics alone. It cannot, however, fix healed slices that are anatomically inconsistent with each other, and it blurs small lesions further. We apply it to every method, ours included. Our claim holds only if the gap survives this post-processing.

Why not just go fully 3D? Native 3D is the principled fix, and it is the strongest baseline above. The case against it is budgetary. Memory for 3D grows with volume, so fitting a whole scan forces a larger compression stride, and S4ND and LIP show that coarse strides wash out small targets. If a 3D model at matched memory keeps recall on small lesions, our method has no advantage, and the size-stratified comparison will show it.

Decisive Experiments

We train on healthy UK Biobank MRI and evaluate on BRATS, WMH and MSLUB. For CT we use the head CT hemorrhage data from the source diffusion paper. Metrics are voxel Dice and area under the precision–recall curve, lesion-wise recall stratified by diameter, and a cross-slice coherence score. The coherence score counts false-positive connected components on healthy volumes that span only one slice, and measures the mean depth-gradient of the residual map.

Three results decide the proposal. The first is the grid-spacing sweep: a clear interior optimum supports the capacity account, and a monotone curve refutes it. The second is size-stratified recall against 3D diffusion at matched memory: our advantage should be concentrated below a few stride units and vanish for large tumors. The third is the synthetic injection test for leakage from the first lesion-bearing slice into later healed slices: our method should show near-zero leakage, and the raw-neighbor 2.5D variant should not.

A secondary transfer test uses LUNA16 nodules, with scans selected per the LUNA16 challenge, as small, roughly spherical anomalies. Fully supervised S4ND reaches a competition performance metric (CPM) of 0.897 there. That number frames how much ground unsupervised healing gives up; we do not expect to match it.