Exploring the tradeoff between data privacy and utility with a clinical data analysis use case

Eunyoung Im, Hyeoneui Kim, Hyungbok Lee, Xiaoqian Jiang, Ju Han KimBMC Medical Informatics Decis. Mak 2024 · OpenReview

TL;DR

This study explores the critical balance between data privacy and utility in clinical datasets by de-identifying patient data using various methods and evaluating the impact on a predictive model for emergency department length of stay. The findings highlight that while de-identification effectively reduces re-identification risk, it often compromises data utility by suppressing records and variables, underscoring the need for careful consideration of intended data use when choosing de-identification strategies.

What Problem Does This Paper Address?

The increasing volume and value of clinical data for research and healthcare improvement are met with a growing imperative to protect patient privacy. De-identification, a common technique to mask sensitive information, can inadvertently degrade the usefulness of the data for analysis. However, there's a lack of clear understanding on how different de-identification methods specifically impact the outcomes of real-world clinical data analyses. This paper aims to bridge this gap by systematically investigating the trade-off between privacy protection and data utility in a practical clinical use case, seeking to identify feasible strategies for balancing these competing needs.

Methods

The researchers employed a data analysis use case: predicting the length of stay (LOS) in an emergency department (ED) using logistic regression. They started with an original clinical dataset and generated 19 de-identified versions using ARX, an open-source anonymization tool. These de-identification scenarios varied in the types of quasi-identifiers (e.g., sex, age, sending hospital, primary diagnosis) and sensitive information treated, and applied privacy-preserving techniques like K-anonymity, L-diversity, and T-closeness. The study then compared the variable distributions and the predictive performance of logistic regression models built on both the original and de-identified datasets. Key metrics used to evaluate the de-identification process included re-identification risk reduction, ARX utility scores, Earth Mover's Distance (EMD), and dataset retention ratios.

Datasets

The study utilized a clinical dataset comprising 1,155 patient cases from the ED of an academic medical center in Seoul, South Korea, collected between January 2019 and December 2019. This dataset contained patient demographics, clinical details, and transfer-related information. The researchers generated 19 de-identified versions of this dataset for their analysis.

Key Insights and Ideas

  • De-identification is a double-edged sword: While effective at reducing re-identification risk, the process inherently leads to data loss, impacting the utility of the dataset for downstream analysis.
  • No one-size-fits-all: The "optimal" de-identification strategy is highly dependent on the specific data analysis task. What works for one prediction model might not work for another.
  • Quasi-identifiers matter significantly: Variables that can uniquely link individuals when combined (like sending hospital and primary diagnosis) are crucial for de-identification but also represent significant points of data loss when masked.
  • Simpler de-identification can preserve utility: Datasets de-identified using only sex and age as quasi-identifiers showed the highest utility (minimal information loss, highest retention ratio), but this might not be sufficient for robust privacy protection against more sophisticated attacks.
  • The ARX utility score is a useful indicator: This metric showed a significant correlation with re-identification risk reduction, suggesting it can help quantify the privacy-utility trade-off.

Key Results

  • All 19 de-identification scenarios significantly reduced re-identification risk.
  • However, de-identification processes led to record suppression and the masking of predictor variables, compromising dataset utility.
  • Models built on datasets with minimal de-identification (using only sex and age as quasi-identifiers) showed the highest similarity to the original dataset, indicating superior utility.
  • The predictive performance (AUC) of the logistic regression models was generally comparable between the original and most de-identified datasets, with significant differences only observed in scenarios with extensive data masking (datasets 7 and 8), which retained fewer predictors.
  • A significant correlation was found between re-identification reduction rates and ARX utility scores, indicating that higher privacy often comes at the cost of lower utility.

Limitations and Future Work

The study acknowledges its limitations:

  • Single use case: The findings are specific to predicting ED LOS and may not generalize to all clinical data analysis tasks.
  • Single hospital data: The dataset was sourced from one hospital, potentially limiting the generalizability of the results.
  • Limited de-identification scenarios: The study did not explore the full spectrum of complex de-identification configurations.

Future work could involve:

  • Investigating a broader range of de-identification scenarios and data analysis tasks.
  • Exploring more advanced privacy-preserving techniques.
  • Incorporating input from data users to better understand their needs and guide the de-identification process towards a more effective compromise between privacy and utility.

Summary updated

Abstract

Securing adequate data privacy is critical for the productive utilization of data. De-identification, involving masking or replacing specific values in a dataset, could damage the dataset’s utility. However, finding a reasonable balance between data privacy and utility is not straightforward. Nonetheless, few studies investigated how data de-identification efforts affect data analysis results. This study aimed to demonstrate the effect of different de-identification methods on a dataset’s utility with a clinical analytic use case and assess the feasibility of finding a workable tradeoff between data privacy and utility. Predictive modeling of emergency department length of stay was used as a data analysis use case. A logistic regression model was developed with 1155 patient cases extracted from a clinical data warehouse of an academic medical center located in Seoul, South Korea. Nineteen de-identified datasets were generated based on various de-identification configurations using ARX, an open-source software for anonymizing sensitive personal data. The variable distributions and prediction results were compared between the de-identified datasets and the original dataset. We examined the association between data privacy and utility to determine whether it is feasible to identify a viable tradeoff between the two. All 19 de-identification scenarios significantly decreased re-identification risk. Nevertheless, the de-identification processes resulted in record suppression and complete masking of variables used as predictors, thereby compromising dataset utility. A significant correlation was observed only between the re-identification reduction rates and the ARX utility scores. As the importance of health data analysis increases, so does the need for effective privacy protection methods. While existing guidelines provide a basis for de-identifying datasets, achieving a balance between high privacy and utility is a complex task that requires understanding the data’s intended use and involving input from data users. This approach could help find a suitable compromise between data privacy and utility.
BibTeX
@inproceedings{im2024exploring,
  author = {Eunyoung Im and Hyeoneui Kim and Hyungbok Lee and Xiaoqian Jiang and Ju Han Kim},
  title = {{Exploring the tradeoff between data privacy and utility with a clinical data analysis use case}},
  booktitle = {BMC Medical Informatics Decis. Mak},
  year = {2024},
  url = {https://openreview.net/forum?id=jso2CRDQtW},
}
Figure