Abstract
Understanding and addressing unfairness in LLMs are crucial for responsible AI deployment. However, there is a limited availability of quantitative analyses and in-depth studies regarding fairness evaluations in LLMs, especially when applying LLMs to high-stakes fields. This work aims to fill this gap by providing a systematic evaluation of the effectiveness and fairness of LLMs using ChatGPT as a study case. We focus on assessing ChatGPT's performance in high-stakes fields including education, criminology, finance, and healthcare. To make a thorough evaluation, we consider both group fairness and individual fairness and we also observe the disparities in ChatGPT's outputs under a set of biased or unbiased prompts. This work contributes to a deeper understanding of LLMs' fairness performance, facilitates bias mitigation, and fosters the development of responsible artificial intelligence systems.
Introduction
As Large Language Models (LLMs) like ChatGPT transition from conversational agents to decision-support tools, their fairness in high-stakes environments becomes a critical concern. While previous research has often focused on linguistic biases or toxic outputs, there is a pressing need for quantitative assessments in domains where algorithmic decisions directly impact human lives, such as law, medicine, and finance. This study systematically evaluates ChatGPT’s fairness and effectiveness across four high-stakes benchmarks, comparing it against traditional machine learning baselines.
Methodology
The evaluation compares ChatGPT (gpt-3.5-turbo) with two traditional "small" models: Logistic Regression (LR) and a Multi-Layer Perceptron (MLP). A central component of the study is the use of Prompt Engineering to investigate how different instructions and examples influence model behavior.
The researchers designed eight distinct prompt types:
- Unbiased Prompts (1-4): These prompts utilized balanced examples, omitted sensitive features, or employed counterfactual scenarios. Prompt 4 specifically instructed the model to "be unbiased."
- Biased Prompts (5-8): These prompts intentionally included examples that associated specific demographic groups with certain outcomes to test if the model would adopt these biases.
Fairness was measured using standard metrics, including Statistical Parity, Equal Opportunity, and Counterfactual Fairness (the consistency of predictions when only a sensitive attribute is changed).
Datasets
The study utilized four diverse benchmarks:
- PISA (Education): Predicting student reading levels (Sensitive feature: Gender).
- COMPAS (Criminology): Predicting recidivism risk (Sensitive feature: Race).
- German Credit (Finance): Assessing credit risk (Sensitive feature: Gender).
- Heart Disease (Healthcare): Predicting heart disease presence (Sensitive feature: Gender).
Results and Discussion
Effectiveness vs. Fairness
ChatGPT generally exhibited superior group fairness compared to traditional models. For instance, in the PISA dataset, ChatGPT achieved a Statistical Parity Difference ($D_{SP}$) as low as 0.08%, significantly lower than the 33.53% observed in the LR model. However, this improved fairness did not always translate to higher accuracy. In the finance domain (German Credit), ChatGPT’s accuracy dropped to approximately 53%, while the MLP baseline maintained 77%, suggesting that LLMs may struggle with complex tabular data compared to specialized models.
Individual and Counterfactual Fairness
One of the most significant findings was ChatGPT's performance in individual fairness. It was far more consistent than traditional models when presented with nearly identical cases that differed only by a sensitive attribute. On the COMPAS dataset, ChatGPT's "change rate" (the frequency of prediction flips when race was changed) was approximately 5%, whereas the MLP model's change rate reached nearly 27%.
The Impact of Prompting
The model proved highly sensitive to prompt variations. Interestingly, explicit instructions to "be unbiased" (Prompt 4) were not always the most effective strategy; providing balanced, few-shot examples often yielded better fairness results. This highlights the "black box" nature of LLMs, where subtle changes in phrasing can lead to significant shifts in output fairness and accuracy.
Conclusion
While ChatGPT shows promise in providing fairer outcomes—particularly regarding individual fairness—its sensitivity to prompts and occasional performance lags in specific domains like finance warrant caution. The study underscores that LLMs are not yet a "drop-in" replacement for specialized models in high-stakes fields but rather tools that require careful calibration and oversight to ensure responsible deployment.
Summary updated