To be or not to be, when synthetic data meet clinical pharmacology: A focused study on pharmacogenetics

Jean‐Baptiste Woillard, Clément Benoist, Alexandre Destère, Marc Labriffe, Giulia Marchello, Julie Josse, Pierre MarquetCPT 2024 · DOI

TL;DR

This study explores the use of synthetic data generation methods (CT-GAN, TVAE, and Avatar) for pharmacogenetics research, aiming to balance data utility and privacy. The Avatar method, particularly with a specific parameter (k=10), and CT-GAN showed promising results in replicating key findings from a small, real-world dataset, suggesting their potential for enabling open science and data sharing in this field.

What Problem Does This Paper Address?

Sharing sensitive patient data in medical research, especially in pharmacology, is crucial for advancing drug discovery and personalized medicine. However, strict privacy regulations like GDPR make data sharing challenging. Synthetic data offers a potential solution by creating artificial datasets that mimic real data's statistical properties without revealing individual patient information. This paper addresses the question of how well different synthetic data generation methods perform when applied to small, real-world pharmacogenetic datasets, a common scenario in clinical research where data is often scarce and privacy concerns are paramount. The authors specifically investigate the trade-off between the utility of synthetic data (how well it preserves the original data's analytical properties) and its privacy guarantees.

Methods

The researchers implemented and compared three synthetic data generation methods:

  • CT-GAN (Conditional Generative Adversarial Network): A deep learning approach that uses a generator and a discriminator to create synthetic data. The "conditional" aspect allows for guided generation.
  • TVAE (Tabular Variational Autoencoder): Another deep learning method that learns a compressed representation of the data and then reconstructs it, introducing randomness for variation.
  • Avatar: A novel method that reduces data dimensionality, uses a nearest neighbors approach in a latent space to create local models, and generates synthetic data based on these models. A key hyperparameter for Avatar is 'k', representing the number of nearest neighbors used.

The study also explored data augmentation (increasing the dataset size) and the effect of running algorithms multiple times to improve result stability. They evaluated the synthetic data using various metrics, including fidelity (how well the synthetic data represents the original data's statistical properties and relationships) and privacy (how difficult it is to re-identify individuals from the synthetic data).

Datasets

The study utilized a previously published pharmacogenetic dataset consisting of 253 patients. This dataset was derived from a study linking Single Nucleotide Polymorphisms (SNPs) in kidney donors with clinical outcomes in renal transplant recipients treated with cyclosporine. The dataset included continuous variables (e.g., age, time to event) and categorical variables (e.g., haplotype, sex).

Key Insights and Ideas

  • The "sweet spot" for Avatar: The Avatar method's performance was highly dependent on the 'k' parameter. A value of k=10 (using 10 nearest neighbors) appeared to strike the best balance between data utility and privacy preservation.
  • Deep learning methods have nuances: While CT-GAN showed good overall performance, TVAE lagged behind in both utility and privacy. The study also noted that deep learning methods might be more suited for larger datasets, with their performance on smaller datasets being less predictable.
  • Running algorithms multiple times matters: For small datasets, repeatedly generating synthetic data and aggregating results (or running the algorithm multiple times) significantly improved the stability and reliability of the findings, particularly for hazard ratio estimations.
  • Privacy vs. Utility is a real trade-off: While synthetic data aims to achieve both, the study found that methods offering stronger privacy guarantees (like CT-GAN and TVAE) sometimes produced synthetic data that was more easily distinguishable from the original, impacting utility. Conversely, Avatar with a well-tuned 'k' offered a good balance.
  • Avatar's unique approach: The Avatar method's local modeling approach, while potentially shrinking the overall data distribution, showed promise in preserving key analytical relationships, especially when compared to other methods.

Key Results

  • Best overall performance: CT-GAN and Avatar with k=10 demonstrated the best combined data utility and privacy preservation.
  • Hazard Ratio (HR) estimation: Avatar with k=10 produced HR estimates closest to the original data. CT-GAN slightly underestimated HR, while TVAE showed significant deviations.
  • Variable selection: For predictive models, Avatar (especially with k=5 and k=20) and CT-GAN were generally successful in identifying the key variable (haplotype) that was significant in the original analysis. However, augmented datasets sometimes introduced "false-positive" associations.
  • Privacy metrics: CT-GAN and TVAE showed higher privacy scores (DCR and NNDR) but also lower utility. Avatar's privacy increased with higher 'k' values.
  • Discrimination analysis: The AUC-ROC for distinguishing between original and synthetic data was around 0.5 for Avatar (indicating high similarity), while CT-GAN and TVAE were around 0.3 (suggesting they are more distinguishable).
  • SurvVAE's limitations: The SurvVAE method consistently produced disappointing results, with Kaplan-Meier curves deviating significantly from the original data and HR estimates being underestimated.

Limitations and Future Work

  • Generalizability: The authors acknowledge that their findings are specific to the pharmacogenetic dataset used and may not generalize to other types of pharmacological data (e.g., longitudinal or population pharmacokinetic data).
  • Small dataset focus: While the study highlights the challenges and potential of synthetic data for small datasets, the methods themselves are often designed for larger datasets.
  • Differential Privacy (DP): The study did not initially generate data under formal DP, and a posteriori application of DP significantly reduced data utility, highlighting the inherent trade-off.
  • Future Directions: The authors suggest further investigation into applying these synthetic data generation methods to other types of pharmacological data and analyses. They also emphasize the ongoing work within their consortium on population pharmacokinetics and longitudinal datasets. Future research could also explore broader applications of synthetic data in machine learning for augmenting datasets, enhancing model training, and strengthening privacy.

Summary updated

Abstract

The use of synthetic data in pharmacology research has gained significant attention due to its potential to address privacy concerns and promote open science. In this study, we implemented and compared three synthetic data generation methods, CT-GAN, TVAE, and a simplified implementation of Avatar, for a previously published pharmacogenetic dataset of 253 patients with one measurement per patient (non-longitudinal). The aim of this study was to evaluate the performance of these methods in terms of data utility and privacy trade off. Our results showed that CT-GAN and Avatar used with k = 10 (number of patients used to create the local model of generation) had the best overall performance in terms of data utility and privacy preservation. However, the TVAE method showed a relatively lower level of performance in these aspects. In terms of Hazard ratio estimation, Avatar with k = 10 produced HR estimates closest to the original data, whereas CT-GAN slightly underestimated the HR and TVAE showed the most significant deviation from the original HR. We also investigated the effect of applying the algorithms multiple times to improve results stability in terms of HR estimation. Our findings suggested that this approach could be beneficial, especially in the case of small datasets, to achieve more reliable and robust results. In conclusion, our study provides valuable insights into the performance of CT-GAN, TVAE, and Avatar methods for synthetic data generation in pharmacogenetic research. The application to other type of data and analyses (data driven) used in pharmacology should be further investigated.
BibTeX
@inproceedings{woillard2024be,
  author = {Jean‐Baptiste Woillard and Clément Benoist and Alexandre Destère and Marc Labriffe and Giulia Marchello and Julie Josse and Pierre Marquet},
  title = {{To be or not to be, when synthetic data meet clinical pharmacology: A focused study on pharmacogenetics}},
  booktitle = {CPT},
  year = {2024},
  doi = {10.1002/psp4.13240},
}
Figure