Evaluating and Improving the Performance and Racial Fairness of Algorithms for GFR Estimation

Linying Zhang, Lauren R. Richter, T. Y. Kim, George Hripcsak

Abstract

Data-driven clinical prediction algorithms are used widely by clinicians. Understanding what factors can impact the performance and fairness of data-driven algorithms is an important step towards achieving equitable healthcare. To investigate the impact of modeling choices on the algorithmic performance and fairness, we make use of a case study to build a prediction algorithm for estimating glomerular filtration rate (GFR) based on the patient's electronic health record (EHR). We compare three distinct approaches for estimating GFR: CKD-EPI equations, epidemiological models, and EHR-based models. For epidemiological models and EHR-based models, four machine learning models of varying computational complexity (i.e., linear regression, support vector machine, random forest regression, and neural network) were compared. Performance metrics included root mean squared error (RMSE), median difference, and the proportion of GFR estimates within 30% of the measured GFR value (P30). Differential performance between non-African American and African American group was used to assess algorithmic fairness with respect to race. Our study showed that the variable race had a negligible effect on error, accuracy, and differential performance. Furthermore, including more relevant clinical features (e.g., common comorbidities of chronic kidney disease) and using more complex machine learning models, namely random forest regression, significantly lowered the estimation error of GFR. However, the difference in performance between African American and non-African American patients did not decrease, where the estimation error for African American patients remained consistently higher than non-African American patients, indicating that more objective patient characteristics should be discovered and included to improve algorithm performance.

Introduction

Glomerular Filtration Rate (GFR) is the clinical gold standard for measuring kidney function, yet direct measurement is invasive and impractical for routine care. Clinicians typically rely on estimated GFR (eGFR) derived from serum creatinine levels. Historically, the standard CKD-EPI equation utilized a "race modifier" that assigned higher eGFR values to Black patients. This practice has faced significant scrutiny for lacking a biological basis and potentially delaying critical care, such as specialist referrals or transplants, for Black individuals.

As the medical community transitions toward "race-free" equations, it is vital to recognize that simply removing a variable does not guarantee algorithmic fairness. This study investigates whether replacing race with a broader set of clinical data from Electronic Health Records (EHRs) and employing sophisticated machine learning models can enhance both the accuracy of these estimates and the equity of outcomes across racial groups.

Methods

The study utilized de-identified EHR data from Columbia University Irving Medical Center (CUIMC) spanning 1980 to 2021. The cohort consisted of 2,352 non-pregnant adult patients, including 610 African American and 1,742 non-African American individuals. The "measured GFR" (mGFR) benchmark was approximated using 24-hour urinary creatinine clearance.

Researchers compared three distinct modeling approaches, each evaluated both with and without race as a predictor:

  1. CKD-EPI Equations: The traditional baseline using age, sex, and creatinine.
  2. Epidemiological Models: Included additional clinical variables: hypertension, type 2 diabetes, height, and weight.
  3. EHR-based Models: Incorporated a patient’s full medical history (diagnoses, medications, labs) from the preceding year.

For the latter two categories, four machine learning architectures were tested: Linear Regression (LR), Support Vector Machines (SVM), Random Forest (RF), and Artificial Neural Networks (NN).

Study workflow including cohort building, evaluated algorithms, and evaluation metrics.

Results

The inclusion of more relevant clinical features, such as comorbidities, rendered the "race" variable negligible in its effect on model performance. This suggests that the "race effect" in traditional equations likely served as a proxy for underlying health disparities and physiological differences that are more accurately captured by objective clinical data.

Transitioning from simple linear equations to Random Forest models significantly reduced estimation error (RMSE) for all patient groups. For African American patients, RMSE dropped from 33.0 (standard CKD-EPI) to 23.5 when using the EHR-based Random Forest model. However, a persistent performance gap remained: African American patients consistently exhibited higher estimation errors—approximately 4 mL/min/1.73m² higher—than non-African American patients across all tested models.

Comparison of performance metrics (RMSE, Median Difference, P30 Accuracy) across different models and racial groups.

Discussion and Conclusion

While leveraging rich EHR data and complex models like Random Forest significantly improves GFR estimation accuracy, it does not eliminate the performance gap between racial groups. This persistent disparity may stem from sample imbalance in the training data or the absence of more objective biological markers.

True fairness in clinical algorithms requires more than just removing race as a variable; it necessitates ensuring an equal distribution of error across all groups. Future research should focus on identifying objective patient characteristics, such as genetic or proteomic markers, and utilizing more diverse datasets to close the performance gap and move beyond race as a social construct in clinical medicine.

Summary updated

Abstract

Data-driven clinical prediction algorithms are used widely by clinicians. Understanding what factors can impact the performance and fairness of data-driven algorithms is an important step towards achieving equitable healthcare. To investigate the impact of modeling choices on the algorithmic performance and fairness, we make use of a case study to build a prediction algorithm for estimating glomerular filtration rate (GFR) based on the patient's electronic health record (EHR). We compare three distinct approaches for estimating GFR: CKD-EPI equations, epidemiological models, and EHR-based models. For epidemiological models and EHR-based models, four machine learning models of varying computational complexity (i.e., linear regression, support vector machine, random forest regression, and neural network) were compared. Performance metrics included root mean squared error (RMSE), median difference, and the proportion of GFR estimates within 30% of the measured GFR value (P30). Differential performance between non-African American and African American group was used to assess algorithmic fairness with respect to race. Our study showed that the variable race had a negligible effect on error, accuracy, and differential performance. Furthermore, including more relevant clinical features (e.g., common comorbidities of chronic kidney disease) and using more complex machine learning models, namely random forest regression, significantly lowered the estimation error of GFR. However, the difference in performance between African American and non-African American patients did not decrease, where the estimation error for African American patients remained consistently higher than non-African American patients, indicating that more objective patient characteristics should be discovered and included to improve algorithm performance.
BibTeX
@misc{zhang2024evaluating,
  author = {Linying Zhang and Lauren R. Richter and T. Y. Kim and George Hripcsak},
  title = {{Evaluating and Improving the Performance and Racial Fairness of Algorithms for GFR Estimation}},
  year = {2024},
  doi = {10.1101/2024.01.07.24300943},
  howpublished = {medRxiv},
}
Figure