Abstract
Predictive modeling in healthcare is often hindered by data privacy regulations and the lack of ground truth for the theoretical maximum predictability of clinical outcomes from medical imaging. This paper introduces a novel framework that bridges generative image models with statistical longitudinal models to create large-scale synthetic datasets with controlled correlations. By linking the latent space of a Variational Autoencoder (VAE) to the random effects of a Nonlinear Mixed-Effects (NLME) model, researchers can precisely define the predictive signal within synthetic medical images. We demonstrate this framework using a dataset of 1.1 million synthetic Optical Coherence Tomography (OCT) scans paired with longitudinal trajectories, providing a rigorous benchmark for evaluating the performance of predictive AI models under varying noise conditions.
Introduction
In medical AI, the scarcity of high-quality, paired datasets—linking medical images to long-term patient outcomes—remains a significant barrier due to privacy regulations like HIPAA and GDPR. Furthermore, even with access to real-world data, the inherent "ground truth" of how much information an image actually contains regarding future disease progression is unknown. This makes it difficult to determine if a model's poor performance is due to architectural flaws or a lack of predictive signal in the data itself.
The proposed framework addresses these challenges by generating "digital twins": synthetic patient data with perfectly controlled correlations between baseline image covariates and longitudinal outcomes.
Methods
The framework employs a three-stage pipeline to generate realistic synthetic data:
- Latent Bridge (VAE): A Variational Autoencoder (VAE) is trained on real 2D Optical Coherence Tomography (OCT) scans to compress images into a low-dimensional latent vector ($z$).
- Image Generation: Stable Diffusion 1.5 is adapted to generate realistic medical scans conditioned directly on the VAE’s latent vectors, preserving structural integrity without relying on text prompts.
- Longitudinal Simulation: Nonlinear Mixed-Effects (NLME) models simulate patient trajectories over time. The "link" between the image and the outcome is established by mapping specific dimensions of the VAE latent vector to the "random effects" ($\eta$) in the NLME model.
To control task difficulty, varying levels of Gaussian noise ($\sigma^2$) are injected into the link between the latent vector and the random effects, allowing for precise adjustment of the explained variance.
Datasets
The models were trained on a public dataset of 109,309 2D OCT scan slices. The resulting synthetic release includes 1.1 million images, each paired with five longitudinal trajectories representing different noise levels, totaling 5.5 million data pairs. This massive dataset serves as a robust playground for benchmarking predictive models.
Results and Validation
The framework was validated by training a ResNet18 to recover the hidden random effects from synthetic images. The performance was evaluated across different noise levels ($\sigma^2 \in \{0, 1, 9, 18, 49\}$).
As shown in the results table, predictive accuracy ($R^2$) decreases as the noise level increases, aligning with the theoretical maximums defined by the framework. Notably, even in high-noise scenarios where the image explains only a small fraction of the variance, the model successfully extracted a meaningful signal, often reaching a significant portion of the theoretical maximum predictability.
Discussion and Conclusion
While synthetic data serves as a proxy and may lack some microscopic biomarkers found in real clinical scans, this framework provides a vital "sanity check" for healthcare AI. It allows researchers to stress-test models in scenarios with weak predictive signals, which is common in real-world clinical settings. Future work could extend this approach to other modalities, such as linking MRI scans to neurodegenerative disease progression or genomic data to clinical outcomes. By open-sourcing the code and the 5.5-million-pair dataset, the authors provide a foundational resource for the development of more robust and interpretable predictive models in medicine.
Data and Code Availability: * Dataset: https://doi.org/10.57967/hf/2089 * Code: https://github.com/Deltadahl/Image-Longitudinal
Summary updated