A Framework for Evaluating Predictive Models Using Synthetic Image Covariates and Longitudinal Data

Simon Deltadahl, Andreu Vall, Vijay Ivaturi, Niklas KorsboarXiv 2024 · DOI · arXiv

Abstract

Predictive modeling in healthcare is often hindered by data privacy regulations and the lack of ground truth for the theoretical maximum predictability of clinical outcomes from medical imaging. This paper introduces a novel framework that bridges generative image models with statistical longitudinal models to create large-scale synthetic datasets with controlled correlations. By linking the latent space of a Variational Autoencoder (VAE) to the random effects of a Nonlinear Mixed-Effects (NLME) model, researchers can precisely define the predictive signal within synthetic medical images. We demonstrate this framework using a dataset of 1.1 million synthetic Optical Coherence Tomography (OCT) scans paired with longitudinal trajectories, providing a rigorous benchmark for evaluating the performance of predictive AI models under varying noise conditions.

Introduction

In medical AI, the scarcity of high-quality, paired datasets—linking medical images to long-term patient outcomes—remains a significant barrier due to privacy regulations like HIPAA and GDPR. Furthermore, even with access to real-world data, the inherent "ground truth" of how much information an image actually contains regarding future disease progression is unknown. This makes it difficult to determine if a model's poor performance is due to architectural flaws or a lack of predictive signal in the data itself.

The proposed framework addresses these challenges by generating "digital twins": synthetic patient data with perfectly controlled correlations between baseline image covariates and longitudinal outcomes.

Methods

The framework employs a three-stage pipeline to generate realistic synthetic data:

  1. Latent Bridge (VAE): A Variational Autoencoder (VAE) is trained on real 2D Optical Coherence Tomography (OCT) scans to compress images into a low-dimensional latent vector ($z$).
  2. Image Generation: Stable Diffusion 1.5 is adapted to generate realistic medical scans conditioned directly on the VAE’s latent vectors, preserving structural integrity without relying on text prompts.
  3. Longitudinal Simulation: Nonlinear Mixed-Effects (NLME) models simulate patient trajectories over time. The "link" between the image and the outcome is established by mapping specific dimensions of the VAE latent vector to the "random effects" ($\eta$) in the NLME model.
Overview of the synthetic data generation pipeline, linking latent image features to longitudinal outcomes via an NLME model.

To control task difficulty, varying levels of Gaussian noise ($\sigma^2$) are injected into the link between the latent vector and the random effects, allowing for precise adjustment of the explained variance.

Datasets

The models were trained on a public dataset of 109,309 2D OCT scan slices. The resulting synthetic release includes 1.1 million images, each paired with five longitudinal trajectories representing different noise levels, totaling 5.5 million data pairs. This massive dataset serves as a robust playground for benchmarking predictive models.

Results and Validation

The framework was validated by training a ResNet18 to recover the hidden random effects from synthetic images. The performance was evaluated across different noise levels ($\sigma^2 \in \{0, 1, 9, 18, 49\}$).

Performance metrics (MSE and R^2) for predicting random effects across varying noise levels, compared against theoretical maximums.

As shown in the results table, predictive accuracy ($R^2$) decreases as the noise level increases, aligning with the theoretical maximums defined by the framework. Notably, even in high-noise scenarios where the image explains only a small fraction of the variance, the model successfully extracted a meaningful signal, often reaching a significant portion of the theoretical maximum predictability.

Discussion and Conclusion

While synthetic data serves as a proxy and may lack some microscopic biomarkers found in real clinical scans, this framework provides a vital "sanity check" for healthcare AI. It allows researchers to stress-test models in scenarios with weak predictive signals, which is common in real-world clinical settings. Future work could extend this approach to other modalities, such as linking MRI scans to neurodegenerative disease progression or genomic data to clinical outcomes. By open-sourcing the code and the 5.5-million-pair dataset, the authors provide a foundational resource for the development of more robust and interpretable predictive models in medicine.

Data and Code Availability: * Dataset: https://doi.org/10.57967/hf/2089 * Code: https://github.com/Deltadahl/Image-Longitudinal

Summary updated

Abstract

We present a novel framework for synthesizing patient data with complex covariates (e.g., eye scans) paired with longitudinal observations (e.g., visual acuity over time), addressing privacy concerns in healthcare research. Our approach introduces controlled association in latent spaces generating each data modality, enabling the creation of complex covariate-longitudinal observation pairs. This framework facilitates the development of predictive models and provides openly available benchmarking datasets for healthcare research. We demonstrate our framework using optical coherence tomography (OCT) scans, though it is applicable across domains. Using 109,309 2D OCT scan slices, we trained an image generative model combining a variational autoencoder and a diffusion model. Longitudinal observations were simulated using a nonlinear mixed effect (NLME) model from a low-dimensional space of random effects. We generated 1.1M OCT scan slices paired with five sets of longitudinal observations at controlled association levels (100%, 50%, 10%, 5.26%, and 2% of between-subject variability). To assess the framework, we modeled synthetic longitudinal observations with another NLME model, computed empirical Bayes estimates of random effects, and trained a ResNet to predict these estimates from synthetic OCT scans. We then incorporated ResNet predictions into the NLME model for patient-individualized predictions. Prediction accuracy on withheld data declined as intended with reduced association between images and longitudinal measurements. Notably, in all but the 2% case, we achieved within 50% of the theoretical best possible prediction on withheld data, demonstrating our ability to detect even weak signals. This confirms the effectiveness of our framework in generating synthetic data with controlled levels of association, providing a valuable tool for healthcare research.
BibTeX
@misc{deltadahl2024framework,
  author = {Simon Deltadahl and Andreu Vall and Vijay Ivaturi and Niklas Korsbo},
  title = {{A Framework for Evaluating Predictive Models Using Synthetic Image Covariates and Longitudinal Data}},
  year = {2024},
  eprint = {2410.16177},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.16177},
}
Figure