Conditional Recurrent Flow: Conditional Generation of Longitudinal Samples with Applications to Neuroimaging

Seong Jae Hwang, Zirui Tao, Won Hwa Kim, Vikas SinghOpenReview 2018 · DOI · OpenReview

Abstract

We develop a conditional generative model for longitudinal image datasets based on sequential invertible neural networks. Longitudinal image acquisitions are common in various scientific and biomedical studies where often each image sequence sample may also come together with various secondary (fixed or temporally dependent) measurements. The key goal is not only to estimate the parameters of a deep generative model for the given longitudinal data, but also to enable evaluation of how the temporal course of the generated longitudinal samples are influenced as a function of induced changes in the (secondary) temporal measurements (or events). Our proposed formulation incorporates recurrent subnetworks and temporal context gating, which provide a smooth transition in a temporal sequence of generated data that can be easily informed or modulated by secondary temporal conditioning variables. We show that the formulation works well despite the smaller sample sizes common in these applications. Our model is validated on two video datasets and a longitudinal Alzheimer's disease (AD) dataset for both quantitative and qualitative evaluations of the generated samples. Further, using our generated longitudinal image samples, we show that we can capture the pathological progressions in the brain that turn out to be consistent with the existing literature, and could facilitate various types of downstream statistical analysis.

Introduction

Most generative models, such as standard GANs or VAEs, are designed for cross-sectional data, generating independent samples. However, many scientific fields, especially medicine, rely on longitudinal studies where the same subject is observed multiple times. This introduces two major challenges: temporal dependency, where data at time $t$ is heavily dependent on time $t-1$, and dynamic conditioning, where researchers need to perform "what-if" scenarios regarding disease progression. For example, predicting how a patient's brain scan might look years later based on a sudden change in cognitive scores.

Methodology: Conditional Recurrent Flow (CRow)

The core innovation is the Conditional Recurrent Flow (CRow), which builds upon Normalizing Flows. Normalizing Flows learn an invertible mapping between a complex data distribution and a simple latent space. CRow makes these flows recurrent through two key mechanisms:

  • Recurrent Sub-networks: Inside the coupling layers, Gated Recurrent Units (GRUs) are embedded to pass a "hidden state" from one time point to the next, providing the model with memory.
  • Temporal Context Gating (TCG): To maintain invertibility, TCG uses the previous hidden state to scale current input via a gating mechanism. This allows conservative adjustments based on history without breaking the mathematical requirements of the flow.
The CRow architecture incorporating recurrent units ($q_1, q_2$) and temporal context gating ($f_{TCG}$).

The model was evaluated on Moving MNIST for qualitative sanity checks and the ADNI/TADPOLE neuroimaging dataset, which includes PET scans and clinical covariates for Alzheimer's progression across 82 brain regions.

Results and Insights

CRow prioritizes temporal smoothness over independence. Unlike conditional INNs (cINNs) that generate time points independently, CRow ensures biologically plausible trajectories. Because the model is invertible, it can perform both generation from a latent code and exact density estimation, allowing it to "flag" unrealistic generated sequences.

Qualitative Smoothness

In Moving MNIST experiments, CRow produced gradual, natural transitions between digits when labels were flipped mid-sequence (e.g., a '5' morphing into a '3'). This contrasts with baseline models that often show abrupt, unrealistic changes.

Qualitative results on Moving MNIST showing smooth transitions between digit labels over time.

Statistical Power in Neuroimaging

Using CRow to generate synthetic populations of "Progressors" vs. "Controls" allowed for enhanced statistical analysis. The model identified 12 Regions of Interest (ROIs) associated with Alzheimer's—significantly more than baseline models. These ROIs, including the hippocampus, amygdala, and putamen, align perfectly with established medical literature on Alzheimer’s pathology.

Conclusion

CRow demonstrates robustness in small-sample regimes typical of neuroimaging. While currently limited by sequence length and feature dimensionality (ROI-based vs. voxel-wise), it provides a powerful framework for simulating complex longitudinal trajectories and capturing real biological signals. Future work will focus on scaling to full-resolution 3D volumes and exploring causal discovery.

Summary updated

Abstract

Generative models using neural network have opened a door to large-scale studies for various application domains, especially for studies that suffer from lack of real samples to obtain statistically robust inference. Typically, these generative models would train on existing data to learn the underlying distribution of the measurements (e.g., images) in latent spaces conditioned on covariates (e.g., image labels), and generate independent samples that are identically distributed in the latent space. Such models may work for cross-sectional studies, however, they are not suitable to generate data for longitudinal studies that focus on "progressive" behavior in a sequence of data. In practice, this is a quite common case in various neuroimaging studies whose goal is to characterize a trajectory of pathologies of a specific disease even from early stages. This may be too ambitious especially when the sample size is small (e.g., up to a few hundreds). Motivated from the setup above, we seek to develop a conditional generative model for longitudinal data generation by designing an invertable neural network. Inspired by recurrent nature of longitudinal data, we propose a novel neural network that incorporates recurrent subnetwork and context gating to include smooth transition in a sequence of generated data. Our model is validated on a video sequence dataset and a longitudinal AD dataset with various experimental settings for qualitative and quantitative evaluations of the generated samples. The results with the AD dataset captures AD specific group differences with sufficiently generated longitudinal samples that are consistent with existing literature, which implies a great potential to be applicable to other disease studies.
BibTeX
@misc{hwang2018conditional,
  author = {Seong Jae Hwang and Zirui Tao and Won Hwa Kim and Vikas Singh},
  title = {{Conditional Recurrent Flow: Conditional Generation of Longitudinal Samples with Applications to Neuroimaging}},
  year = {2018},
  eprint = {1811.09897},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1811.09897},
  howpublished = {OpenReview},
}
Figure