FairMedQA: Benchmarking Bias in Large Language Models for Medical Question Answering

Ying Xiao, Jie Huang, Ruijuan He, Tingting Wang, Mohammad Reza Mousavi, Yepang Liu, Kezhi Li, Zhenpeng Chen, Jie M. Zhang

Abstract

Large language models (LLMs) are approaching expert-level performance in medical question answering (QA), demonstrating strong potential to improve public healthcare. However, underlying biases related to sensitive attributes such as sex and race pose life-critical risks. The extent to which such sensitive attributes affect diagnosis remains an open question and requires comprehensive empirical investigation. Additionally, even the latest Counterfactual Patient Variations (CPV) benchmark can hardly distinguish the bias levels of different LLMs. To further explore these dynamics, we propose a new benchmark, FairMedQA, and benchmark 12 representative LLMs. FairMedQA contains 4,806 counterfactual question pairs constructed from 801 clinical vignettes. Our results reveal substantial accuracy disparity ranging from 3 to 19 percentage points across sensitive demographic groups. Notably, FairMedQA exposes biases that are at least 12 percentage points larger than those identified by the latest CPV benchmark, presenting superior benchmarking sensitivity. Our results underscore an urgent need for targeted debiasing techniques and more rigorous, identity-aware validation protocols before LLMs can be safely integrated into practical clinical decision-support systems.

Introduction

As Large Language Models (LLMs) begin to rival human experts in medical examinations, a critical question arises: are these models equally accurate for all patient populations? While models like GPT-4o show incredible promise for clinical decision support, they risk inheriting and amplifying societal biases present in their training data. Previous attempts to measure this bias, such as the Counterfactual Patient Variations (CPV) benchmark, often relied on simple lexical swaps (e.g., changing "man" to "woman"). Modern, sophisticated models can often bypass such simple tests, masking deeper underlying biases. This paper introduces FairMedQA, a new adversarial benchmark designed to rigorously test the fairness of LLMs across race, sex, and socioeconomic status.

Methodology

To create a more sensitive benchmark, we developed a multi-agent framework to generate adversarial "counterfactual" patient stories. The process involves three primary stages:

  1. Clinical Vignette Neutralization: 801 high-quality clinical vignettes from the USMLE were stripped of demographic markers to establish a neutral baseline.
  2. Adversarial Generation: A multi-agent LLM system was employed to create complex variants. A Generation-Agent develops detailed social backgrounds, a Fusion-Agent integrates these into the medical narrative, and a Validation-Agent ensures the variant successfully triggers biased responses without altering clinical facts.
  3. Human-in-the-Loop Auditing: A team of experts conducted over 340 person-hours of manual review to ensure medical accuracy and logical consistency.
The multi-agent framework for FairMedQA dataset construction, including vignette neutralization, adversarial generation, and human auditing.

This process resulted in the FairMedQA dataset, comprising 4,806 counterfactual question pairs evaluated across three sensitive axes: Race (Black vs. White), Sex (Male vs. Female), and Socioeconomic Status (High vs. Low Income).

Results and Analysis

We benchmarked 12 representative models, including the GPT-4 and GPT-5 series, Claude 3.7/4, Gemini 2.0/2.5, Qwen, and DeepSeek. Our analysis revealed that FairMedQA is 12 percentage points more sensitive at detecting bias than previous benchmarks.

Diagnostic accuracy and fairness metrics (Counterfactual Fairness Rate and Accuracy Disparity) for 12 LLMs across race, sex, and socioeconomic status.

Key Findings

  • Substantial Disparity: Accuracy disparities (AD) across demographic groups ranged from 3% to 19%. While GPT-5 (as named in the study's snapshots) emerged as the most "fair" model with a 3% AD and 94% Counterfactual Fairness Rate (CFR), models like Qwen-3 and DeepSeek-V3.1 exhibited disparities up to 19%.
  • The "Win-Win" Discovery: Contrary to the hypothesized "alignment tax," we found that newer model versions often improved in both diagnostic accuracy and fairness simultaneously. This suggests that superior reasoning capabilities allow models to better distinguish clinical evidence from irrelevant demographic noise.
  • Reasoning Pathways: Bias was observed not just in final answers but in the reasoning process itself. Adversarial contexts can nudge a model down flawed logical paths, causing it to prioritize social stereotypes over clinical facts.

Conclusion and Future Work

FairMedQA exposes significant diagnostic disparities in state-of-the-art LLMs, highlighting the risks of clinical integration without rigorous bias testing. While the dataset currently focuses on binary attributes within a US-centric medical context, it provides a vital foundation for future research. Subsequent work will aim to incorporate non-binary identities, a broader spectrum of ethnicities, and global medical standards to ensure that AI-driven clinical support is equitable for all patient groups.

Summary updated

Abstract

Large language models (LLMs) are approaching expert-level performance in medical question answering (QA), demonstrating strong potential to improve public healthcare. However, underlying biases related to sensitive attributes such as sex and race pose life-critical risks. The extent to which such sensitive attributes affect diagnosis remains an open question and requires comprehensive empirical investigation. Additionally, even the latest Counterfactual Patient Variations (CPV) benchmark can hardly distinguish the bias levels of different LLMs. To further explore these dynamics, we propose a new benchmark, FairMedQA, and benchmark 12 representative LLMs. FairMedQA contains 4,806 counterfactual question pairs constructed from 801 clinical vignettes. Our results reveal substantial accuracy disparity ranging from 3 to 19 percentage points across sensitive demographic groups. Notably, FairMedQA exposes biases that are at least 12 percentage points larger than those identified by the latest CPV benchmark, presenting superior benchmarking sensitivity. Our results underscore an urgent need for targeted debiasing techniques and more rigorous, identity-aware validation protocols before LLMs can be safely integrated into practical clinical decision-support systems.
BibTeX
@misc{xiao2025fairmedqa,
  author = {Ying Xiao and Jie Huang and Ruijuan He and Tingting Wang and Mohammad Reza Mousavi and Yepang Liu and Kezhi Li and Zhenpeng Chen and Jie M. Zhang},
  title = {{FairMedQA: Benchmarking Bias in Large Language Models for Medical Question Answering}},
  year = {2025},
  eprint = {2505.19562},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.19562},
}
Figure