Abstract
We develop a conditional generative model for longitudinal image datasets based on sequential invertible neural networks. Longitudinal image acquisitions are common in various scientific and biomedical studies where often each image sequence sample may also come together with various secondary (fixed or temporally dependent) measurements. The key goal is not only to estimate the parameters of a deep generative model for the given longitudinal data, but also to enable evaluation of how the temporal course of the generated longitudinal samples are influenced as a function of induced changes in the (secondary) temporal measurements (or events). Our proposed formulation incorporates recurrent subnetworks and temporal context gating, which provide a smooth transition in a temporal sequence of generated data that can be easily informed or modulated by secondary temporal conditioning variables. We show that the formulation works well despite the smaller sample sizes common in these applications. Our model is validated on two video datasets and a longitudinal Alzheimer's disease (AD) dataset for both quantitative and qualitative evaluations of the generated samples. Further, using our generated longitudinal image samples, we show that we can capture the pathological progressions in the brain that turn out to be consistent with the existing literature, and could facilitate various types of downstream statistical analysis.
Introduction
Most generative models, such as standard GANs or VAEs, are designed for cross-sectional data, generating independent samples. However, many scientific fields, especially medicine, rely on longitudinal studies where the same subject is observed multiple times. This introduces two major challenges: temporal dependency, where data at time $t$ is heavily dependent on time $t-1$, and dynamic conditioning, where researchers need to perform "what-if" scenarios regarding disease progression. For example, predicting how a patient's brain scan might look years later based on a sudden change in cognitive scores.
Methodology: Conditional Recurrent Flow (CRow)
The core innovation is the Conditional Recurrent Flow (CRow), which builds upon Normalizing Flows. Normalizing Flows learn an invertible mapping between a complex data distribution and a simple latent space. CRow makes these flows recurrent through two key mechanisms:
- Recurrent Sub-networks: Inside the coupling layers, Gated Recurrent Units (GRUs) are embedded to pass a "hidden state" from one time point to the next, providing the model with memory.
- Temporal Context Gating (TCG): To maintain invertibility, TCG uses the previous hidden state to scale current input via a gating mechanism. This allows conservative adjustments based on history without breaking the mathematical requirements of the flow.
The model was evaluated on Moving MNIST for qualitative sanity checks and the ADNI/TADPOLE neuroimaging dataset, which includes PET scans and clinical covariates for Alzheimer's progression across 82 brain regions.
Results and Insights
CRow prioritizes temporal smoothness over independence. Unlike conditional INNs (cINNs) that generate time points independently, CRow ensures biologically plausible trajectories. Because the model is invertible, it can perform both generation from a latent code and exact density estimation, allowing it to "flag" unrealistic generated sequences.
Qualitative Smoothness
In Moving MNIST experiments, CRow produced gradual, natural transitions between digits when labels were flipped mid-sequence (e.g., a '5' morphing into a '3'). This contrasts with baseline models that often show abrupt, unrealistic changes.
Statistical Power in Neuroimaging
Using CRow to generate synthetic populations of "Progressors" vs. "Controls" allowed for enhanced statistical analysis. The model identified 12 Regions of Interest (ROIs) associated with Alzheimer's—significantly more than baseline models. These ROIs, including the hippocampus, amygdala, and putamen, align perfectly with established medical literature on Alzheimer’s pathology.
Conclusion
CRow demonstrates robustness in small-sample regimes typical of neuroimaging. While currently limited by sequence length and feature dimensionality (ROI-based vs. voxel-wise), it provides a powerful framework for simulating complex longitudinal trajectories and capturing real biological signals. Future work will focus on scaling to full-resolution 3D volumes and exploring causal discovery.
Summary updated