Recurrent Adversarial Forecasting of Brain Atrophy as Volumetric Deformation

Forecasting how a patient's brain will shrink over the next few years from their past MRI scans is a natural task for generative models. Each patient's future is uncertain, the scans are 3D, and the useful signal sits in the rate and pattern of change across visits rather than in any single image. The two research lines that should meet here have not. Recurrent GANs, which are generator–discriminator pairs whose networks carry a memory across time steps, model sequential uncertainty well but have only been run on 1D sensor streams and low-resolution video. Volumetric 3D networks capture shape across all three axes but are static and hit GPU memory limits at full resolution. The existing systems for longitudinal brain MRI each give up one side. One processes the brain as independent 2D slices. Another treats time as a single jump in a frozen latent space. A third interpolates deterministically between two known scans.

We propose to forecast atrophy by having a recurrent adversarial model generate deformations, not voxels. A 3D convolutional recurrent generator reads a patient's visit history and samples a sequence of coarse 3D velocity fields. Integrating these fields warps the patient's own last scan forward in time. A recurrent discriminator then judges whether the resulting sequence of local volume-change maps looks like real disease progression. Working in deformation space at coarse resolution sidesteps the memory problem, keeps the patient's anatomy intact by construction, and turns "brain tissue should not grow" into a constraint on a quantity the model actually produces.

Core Claim

The claim has three parts. First, a sampled distribution over future atrophy, conditioned on a patient's full visit history through a recurrent hidden state, gives better calibrated and more accurate regional volume forecasts than models that see only one baseline scan. Second, an adversarial critic that judges sequences of local volume-change maps is what keeps those samples realistic without blurring them. Third, these gains should grow with the number of prior visits, because that is when history carries patient-specific rate information. If the gain does not grow with history length, the recurrence is not earning its place, and the experiments are built to show this.

Relation to Prior Work

The closest work is 4D-DANI-Net. It simulates longitudinal 3D T1 MRI on ADNI with one conditional autoencoder per axial slice position. It stitches slices together with penalties on adjacent-slice intensity differences, adds a separate 3D super-resolution stage, and uses voxel- and region-level losses that force tissue to shrink. This paper motivates our design: its ablation attributes a 58.8% performance gain to the super-resolution stage alone, which suggests slice-wise generation leaves much 3D structure to repair. It also supplies our monotonicity idea. It differs from our proposal in having no 3D operator during progression and no critic over the multi-visit sequence.

Longitudinal Causal Image Synthesis produces full 3D brain MRIs from a 3D StyleGAN. Clinical variables, passed through small fully connected networks, shift its latent code. It reports a mean absolute error of 0.011 at 48 months. Each forecast is a single static shift with no memory of previous visits, which is exactly the regime our recurrence ablation targets.

SVIN supplies the method we reuse: learned 3D displacement fields for medical volumes. It is deterministic and needs both endpoint volumes, so it cannot forecast. A natural-video baseline it compares against dropped to 31.08 PSNR versus SVIN's 41.87 on ACDC, which constrains us away from 2D video-style generators.

On the recurrent side, ForGAN shows that adversarially trained recurrent forecasters cover multi-modal futures where squared-error regression collapses to the mean. CPI-GAN shows that adding a monotonic degradation penalty to a recurrent GAN improves downstream prediction under unseen operating conditions. CR-GAN needs a post-hoc classifier to discard implausible generated sequences, which is a warning about unconstrained recurrent GANs. All three work on 1D sensor vectors.

Finally, the longitudinal MRI foundation-model benchmark finds that slice-based vision-language models order scans correctly but score under 16% on volumetric quantification. That finding motivates measuring progression in native 3D, but it is a reasoning benchmark, not a generative baseline.

Research Program

The generator works at a coarse resolution, for example one quarter of the input resolution. At visit $t$, an encoder built from 3D convolutions reads the current warped scan and the elapsed time $\Delta t$ since the previous visit. A 3D convolutional GRU updates a spatial hidden state $h_t$ from $h_{t-1}$, that encoding, and a noise volume $z_t$. A decoder outputs a stationary velocity field $v_t$, which is upsampled and integrated by scaling-and-squaring into a smooth, invertible deformation. Here "smooth, invertible" (diffeomorphic) means the warp neither tears nor folds tissue. The deformations compose across visits:

$$\phi_t = \phi_{t-1} \circ \exp(v_t), \qquad \hat x_t = x_{\text{last}} \circ \phi_t^{-1},$$

where $x_{\text{last}}$ is the most recent observed scan and $\hat x_t$ is the forecast volume. A small residual intensity channel absorbs contrast changes that warping cannot express.

The local volume change at voxel $p$ is $J_t(p) = \log \det \nabla \phi_t(p)$. It is negative where tissue shrinks and positive where it expands. Using a fixed atlas-registered tissue mask $M_{\text{tissue}}$ and a ventricle/CSF mask $M_{\text{csf}}$, we penalize violations of known biology:

$$\mathcal{L}_{\text{mono}} = \sum_t \Big( \sum_{p \in M_{\text{tissue}}} \max(0, J_t(p) - J_{t-1}(p)) + \sum_{p \in M_{\text{csf}}} \max(0, J_{t-1}(p) - J_t(p)) \Big).$$

The penalty applies to cumulative change, so it allows small healthy fluctuations in a single step only to the extent the data demands.

The discriminator is also a 3D convolutional recurrent network. It sees the sequence of $J_t$ maps together with $\Delta t$, and its job is to tell real progression from generated progression. Real sequences come from registering each patient's actual visits to their own first scan. Judging volume-change maps rather than intensities focuses the critic on the dynamics, meaning where and how fast atrophy happens, rather than on scanner texture. A second, lightweight per-visit 3D image critic keeps the warped volumes sharp.

Training uses each patient's prefix of visits to forecast the remaining ones. Visit counts and spacing vary per patient, so the model must handle irregular intervals natively.

Baselines

The naive reference is the no-change forecast. It predicts that the follow-up equals the last observed scan, and alongside it we fit a per-region linear extrapolation of observed volumes. 4D-DANI-Net reports beating linear mixed-effects models by over 30% in grey-matter volumetric error, so any neural method must clear this floor convincingly.

The strongest existing method on this exact task is 4D-DANI-Net. On ADNI it reports grey-matter volumetric error reduced by over 30% relative to linear mixed-effects models. It also reports that 21 experts identified its synthetic follow-ups with only 59.3% accuracy. We will reimplement it on the identical subject split it describes (9,852 training scans from 876 participants, 1,283 test scans from 170 participants), with the same preprocessing, GPU budget, and hyperparameter search budget as our model. Beating it means lower regional volume error for hippocampus, grey matter, and ventricles at matched forecast horizons, with paired tests across test subjects.

The second strong baseline is the static 3D generator of Longitudinal Causal Image Synthesis, which reports MAE 0.011 at 48 months on ADNI. We will run it on the same split, with its clinical-variable inputs restricted to what our model also sees. Beating it means lower 48-month MAE and lower regional volume error.

The key-ingredient ablations come in three variants, all sharing our 3D deformation backbone. The first replaces the recurrent state with a feedforward network that sees only the last scan. The second replaces it with a feedforward network that sees all prior visits stacked along the channel axis and padded. The third keeps recurrence but removes the sequence critic, training with L1 plus a stochastic latent instead.

Why not just…?

Why not just slice the volume and run a 2D video GAN? Evidence from 4D-DANI-Net argues against it: a slice-wise design needed inter-slice penalties plus a 3D super-resolution stage worth 58.8% of its performance. SVIN's comparison also shows a natural-video model losing over ten PSNR points on cardiac volumes. We do not need a separate 2D baseline because 4D-DANI-Net is the stronger slice-wise system, and we compare against it directly.

Why not just feed all prior visits into a feedforward 3D network? This is the real threat. With ADNI's typically short visit histories, a stacked-input network may match a recurrent one. Our recurrence ablation settles it. We stratify test patients by number of prior visits and check whether the recurrent model's advantage over the stacked feedforward model grows with history length and with interval irregularity. If it does not, we will report that the recurrence is unnecessary and keep the deformation-plus-critic design without it.

Why not just train a deterministic 3D deformation forecaster with L1 loss? A deterministic forecaster can predict the mean atrophy well. ForGAN shows that such models fail to cover multi-modal futures, which is reflected in its undefined KL divergence on Lorenz trajectories. Point accuracy may not separate the two approaches, so we evaluate calibration directly. If a deterministic model with a simple Gaussian error bar is as well calibrated as our sampled distribution, the adversarial component is not justified.

Proposed Experiments

The main experiment uses ADNI on the 4D-DANI-Net split. Each test patient's first $k$ visits are used to forecast the rest. Point accuracy is measured by regional volume error from FreeSurfer-style segmentation of forecast versus real follow-ups for hippocampus, grey matter, and ventricles, together with voxel MAE. Distributional quality is measured by continuous ranked probability score on regional volumes and by empirical coverage of 50% and 90% predictive intervals. Realism is measured by the rate of monotonicity violations in $J_t$ and by a blinded reader study patterned on 4D-DANI-Net's Turing test. Generalization is measured on OASIS-3, used purely for cross-scanner testing. There is no reported result on Lacuna for any baseline on OASIS-3 alone, so we report all methods ourselves.

Several checks are decisive. The recurrent-versus-stacked comparison, stratified by history length, determines whether recurrence matters. The sequence-critic ablation determines whether calibration and sharpness come from the adversary or from the stochastic latent alone. The monotonicity penalty with and without its use determines whether the constraint suppresses the hallucinated trajectories that CR-GAN had to filter post hoc, at what cost to accuracy. A peak-memory and runtime comparison against a voxel-space 3D recurrent generator at full resolution tests the claim that coarse-velocity generation avoids the memory problem.