Fairness of ChatGPT

Yunqi Li, Yongfeng Zhang

Abstract

Understanding and addressing unfairness in LLMs are crucial for responsible AI deployment. However, there is a limited availability of quantitative analyses and in-depth studies regarding fairness evaluations in LLMs, especially when applying LLMs to high-stakes fields. This work aims to fill this gap by providing a systematic evaluation of the effectiveness and fairness of LLMs using ChatGPT as a study case. We focus on assessing ChatGPT's performance in high-stakes fields including education, criminology, finance, and healthcare. To make a thorough evaluation, we consider both group fairness and individual fairness and we also observe the disparities in ChatGPT's outputs under a set of biased or unbiased prompts. This work contributes to a deeper understanding of LLMs' fairness performance, facilitates bias mitigation, and fosters the development of responsible artificial intelligence systems.

Introduction

As Large Language Models (LLMs) like ChatGPT transition from conversational agents to decision-support tools, their fairness in high-stakes environments becomes a critical concern. While previous research has often focused on linguistic biases or toxic outputs, there is a pressing need for quantitative assessments in domains where algorithmic decisions directly impact human lives, such as law, medicine, and finance. This study systematically evaluates ChatGPT’s fairness and effectiveness across four high-stakes benchmarks, comparing it against traditional machine learning baselines.

Methodology

The evaluation compares ChatGPT (gpt-3.5-turbo) with two traditional "small" models: Logistic Regression (LR) and a Multi-Layer Perceptron (MLP). A central component of the study is the use of Prompt Engineering to investigate how different instructions and examples influence model behavior.

Example of a few-shot prompt for the COMPAS dataset used to evaluate ChatGPT's recidivism predictions.

The researchers designed eight distinct prompt types:

  • Unbiased Prompts (1-4): These prompts utilized balanced examples, omitted sensitive features, or employed counterfactual scenarios. Prompt 4 specifically instructed the model to "be unbiased."
  • Biased Prompts (5-8): These prompts intentionally included examples that associated specific demographic groups with certain outcomes to test if the model would adopt these biases.

Fairness was measured using standard metrics, including Statistical Parity, Equal Opportunity, and Counterfactual Fairness (the consistency of predictions when only a sensitive attribute is changed).

Datasets

The study utilized four diverse benchmarks:

  1. PISA (Education): Predicting student reading levels (Sensitive feature: Gender).
  2. COMPAS (Criminology): Predicting recidivism risk (Sensitive feature: Race).
  3. German Credit (Finance): Assessing credit risk (Sensitive feature: Gender).
  4. Heart Disease (Healthcare): Predicting heart disease presence (Sensitive feature: Gender).

Results and Discussion

Effectiveness vs. Fairness

ChatGPT generally exhibited superior group fairness compared to traditional models. For instance, in the PISA dataset, ChatGPT achieved a Statistical Parity Difference ($D_{SP}$) as low as 0.08%, significantly lower than the 33.53% observed in the LR model. However, this improved fairness did not always translate to higher accuracy. In the finance domain (German Credit), ChatGPT’s accuracy dropped to approximately 53%, while the MLP baseline maintained 77%, suggesting that LLMs may struggle with complex tabular data compared to specialized models.

Individual and Counterfactual Fairness

One of the most significant findings was ChatGPT's performance in individual fairness. It was far more consistent than traditional models when presented with nearly identical cases that differed only by a sensitive attribute. On the COMPAS dataset, ChatGPT's "change rate" (the frequency of prediction flips when race was changed) was approximately 5%, whereas the MLP model's change rate reached nearly 27%.

Performance and fairness metrics for the COMPAS dataset across different models and prompt types.

The Impact of Prompting

The model proved highly sensitive to prompt variations. Interestingly, explicit instructions to "be unbiased" (Prompt 4) were not always the most effective strategy; providing balanced, few-shot examples often yielded better fairness results. This highlights the "black box" nature of LLMs, where subtle changes in phrasing can lead to significant shifts in output fairness and accuracy.

Conclusion

While ChatGPT shows promise in providing fairer outcomes—particularly regarding individual fairness—its sensitivity to prompts and occasional performance lags in specific domains like finance warrant caution. The study underscores that LLMs are not yet a "drop-in" replacement for specialized models in high-stakes fields but rather tools that require careful calibration and oversight to ensure responsible deployment.

Summary updated

Abstract

Understanding and addressing unfairness in LLMs are crucial for responsible AI deployment. However, there is a limited availability of quantitative analyses and in-depth studies regarding fairness evaluations in LLMs, especially when applying LLMs to high-stakes fields. This work aims to fill this gap by providing a systematic evaluation of the effectiveness and fairness of LLMs using ChatGPT as a study case. We focus on assessing ChatGPT's performance in high-takes fields including education, criminology, finance and healthcare. To make thorough evaluation, we consider both group fairness and individual fairness and we also observe the disparities in ChatGPT's outputs under a set of biased or unbiased prompts. This work contributes to a deeper understanding of LLMs' fairness performance, facilitates bias mitigation and fosters the development of responsible artificial intelligence systems.
BibTeX
@misc{li2023fairness,
  author = {Yunqi Li and Yongfeng Zhang},
  title = {{Fairness of ChatGPT}},
  year = {2023},
  eprint = {2305.18569},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.18569},
}
Figure