Disparities in seizure outcomes revealed by large language models

Kevin Xie, William K.s. Ojemann, Ryan S. Gallagher, Alfredo Lucas, Chloé E. Hill, Roy H. Hamilton, Kevin B. Johnson, Dan Roth, Brian Litt, Colin A. Ellis

Abstract

We found no evidence that our LLM was intrinsically biased against any demographic group. Seizure freedom extracted by LLM revealed disparities in seizure outcomes across several demographic groups. These findings highlight the critical need to reduce disparities in the care of people with epilepsy.

Introduction

While barriers to healthcare access for marginalized groups are well-documented in the United States, disparities in clinical outcomes—particularly among those already receiving specialty care—remain less understood. In epilepsy, "seizure freedom" is a primary clinical goal, yet this information is often buried within the unstructured text of clinical notes, making large-scale analysis difficult. This study leverages a clinical Large Language Model (LLM) to extract seizure outcomes from over 84,000 patient visits, while simultaneously addressing the potential for algorithmic bias.

Methods

The researchers utilized Clinical_BERT, a transformer-based model pre-trained on clinical text and fine-tuned on 700 manually annotated epilepsy clinic notes. To ensure the model did not introduce its own biases, it was rigorously audited against a validation set of 192 notes annotated in triplicate by humans. Performance was evaluated using accuracy, Positive Class Balance (PCB), and Negative Class Balance (NCB) across various demographic categories.

Statistical analysis was performed using logistic mixed-effects regression models to account for longitudinal patient data (84,675 visits from 25,612 patients at the University of Pennsylvania Health System) while controlling for clinical variables such as the number of anti-seizure medications (ASMs).

Results

Algorithmic Fairness

A critical component of this research was verifying that the LLM performed equitably across all patient groups. The audit revealed no significant differences in model accuracy, PCB, or NCB based on race, ethnicity, sex, age, or socioeconomic status.

Model performance metrics (Accuracy, PCB, NCB) across Race, Age, and Income groups.
Model performance metrics (Accuracy, PCB, NCB) across Ethnicity, Sex, and Insurance types.

This high level of consistency (with p-values for bias tests mostly > 0.80) confirms that the LLM serves as an objective tool for identifying real-world healthcare inequities rather than reflecting algorithmic error.

Clinical Disparities in Seizure Outcomes

The analysis revealed significant disparities in seizure freedom even among patients receiving specialized care:

  • Gender: Female patients had significantly higher odds of experiencing recent seizures compared to male patients (OR 1.33, p = 3x10⁻⁸).
  • Socioeconomic Status: Patients with public insurance (OR 1.53) and those residing in the lowest-income zip codes (OR 1.33) were significantly less likely to achieve seizure freedom.
  • Age: Older adults (65+) were more likely to be seizure-free than younger adults (OR 0.43).
  • Race: While Black patients showed worse outcomes in univariable models, this effect was not significant in multivariable analysis (OR 1.03, p = 0.66) once socioeconomic factors were included, suggesting that economic disparities are a primary driver of racial health gaps.

Discussion

These findings demonstrate that access to specialized academic epilepsy centers is not sufficient to eliminate health disparities. Structural barriers, such as the cost of medications, environmental stressors, and other socioeconomic factors, appear to persist within the clinical environment. The study also highlights the utility of validated LLMs in medical sociology, providing a scalable method to audit clinical outcomes that were previously inaccessible.

Limitations and Future Work

The study is limited by its single-center design and the use of binary gender frameworks within the EHR. Furthermore, while the LLM identifies that disparities exist, further research is required to understand the underlying mechanisms, such as medication adherence and systemic stressors, to develop effective interventions.

Summary updated

Abstract

We found no evidence that our LLM was intrinsically biased against any demographic group. Seizure freedom extracted by LLM revealed disparities in seizure outcomes across several demographic groups. These findings highlight the critical need to reduce disparities in the care of people with epilepsy.
BibTeX
@article{xie2024disparities,
  author = {Kevin Xie and William K.S. Ojemann and Ryan S. Gallagher and Alfredo Lucas and Chloé E. Hill and Roy H. Hamilton and Kevin B. Johnson and Dan Roth and Brian Litt and Colin A. Ellis},
  title = {{Disparities in seizure outcomes revealed by large language models}},
  journal = {J. Am. Medical Informatics Assoc.},
  year = {2024},
  doi = {10.1093/jamia/ocae047},
}
Figure