Confronting LLMs with Traditional ML: Rethinking the Fairness of Large Language Models in Tabular Classifications

Yanchen Liu, Srishti Gautam, Jiaqi Ma, Himabindu Lakkaraju

TL;DR

While Large Language Models (LLMs) are increasingly being adapted for tabular classification tasks, this paper reveals that they inherit severe social biases from their pre-training data, often underperforming traditional machine learning models like Random Forests in fairness metrics. Standard mitigation techniques like data resampling fail to help LLMs, though full fine-tuning shows promise in leveling the fairness playing field.

What Problem Does This Paper Address?

Tabular data is the backbone of high-stakes decision-making systems, from predicting credit risks to evaluating criminal recidivism. Recently, a trend has emerged to serialize tabular data into natural language prompts so that powerful LLMs can perform classification tasks.

However, LLMs are trained on massive internet text corpora rife with human stereotypes. This paper addresses a critical safety gap: Do LLMs import these harmful social biases into tabular classification? If so, how does their fairness compare to traditional machine learning (ML) baselines, and can standard bias-mitigation techniques save them?

Methods

The authors evaluate GPT-3.5 (gpt-3.5-turbo) against two traditional baselines: a Random Forest (RF) and a shallow 3-layer Neural Network (NN).

To feed tabular data into the LLM, they serialize rows into text strings (e.g., "workclass: Private, hours per week: 40, sex: Female..."). They test the LLM across several regimes:

  1. Regular Zero-Shot Prompting: Asking the model for a prediction based purely on the serialized row and task description.
  2. In-Context Learning (ICL): Providing 50 few-shot examples within the prompt before the test instance.
  3. Label-Flipped ICL: Intentionally flipping the target labels of the 50 few-shot examples to see if the model relies on input-label mappings or internal semantic priors.
  4. Fine-Tuning (FT): Training the model on the full dataset via OpenAI's API, including variants using data resampling (oversampling the minority group or undersampling the majority group).

Fairness is evaluated using standard metrics: Statistical Parity (SP) and Equality of Opportunity (EoO).

Datasets

The study utilizes three widely recognized tabular benchmarks for fairness research:

  • Adult Income Dataset: Predicts whether an individual earns >$50k/year. Protected attribute: Sex (Female as the disadvantaged group).
  • German Credit Dataset: Classifies bank customers as good or bad credit risks. Protected attribute: Sex (Female as the marginalized group).
  • COMPAS Dataset: Predicts two-year criminal recidivism. Protected attribute: Race (African-American as the disadvantaged group).

Key Insights and Ideas

  • Biases are Inherent, Not Just Downstream: The unfairness observed in LLMs isn't just a reflection of imbalances in the tabular datasets; it stems directly from the model's pre-training corpus. Even when protected attributes (like race or sex) are completely removed from the prompt, a significant fairness gap remains.
  • The Label-Flipping Paradox: Flipping the labels of few-shot examples drastically reduces the fairness gap, nearly eliminating the bias in some cases. This proves that the LLM's tabular predictions are heavily driven by its internal semantic stereotypes rather than purely learning data correlations. However, this fairness gain comes at the cost of overall predictive accuracy.
  • Resampling Fails for LLMs: In traditional ML, oversampling or undersampling data is a reliable way to balance group representations and improve fairness. For LLMs, resampled fine-tuning yields highly inconsistent results, demonstrating that LLMs process data distributions fundamentally differently than traditional architectures.

Key Results

In zero-shot settings, GPT-3.5 exhibits massive fairness disparities that frequently dwarf traditional ML models. For instance, on the Adult dataset, the Equality of Opportunity (EoO) gap between men and women reached 0.483 for GPT-3.5, compared to just 0.195 for Random Forest.

Figure 1: Fairness Comparison of GPT-3.5 Zero-Shot Prompting with RF and NN. We present the absolute differences in ACC , F1 , SP , and EoO between subgroups of protected attributes across three datasets: “sex” for Adult and German Credit, and “race” for COMPAS. Notable fairness gaps between protected and non-protected subgroups are observed for GPT-3.5 in a zero-shot prompting manner (GPT-3.5 Zero-Shot), which are significantly larger compared to those observed with RF and NN, except for the German Credit dataset.

While regular few-shot learning (ICL) narrows this gap slightly, the disparities remain higher than traditional baselines. However, Regular Fine-Tuning (FT) proved highly effective, dropping the Adult EoO gap to 0.0714—successfully outperforming the Neural Network baseline (0.123).

Figure 3: Fairness Comparison of GPT-3.5 Finetuning with RF and NN Finetuning. We compare the absolute differences in fairness metrics - ACC , F1 , SP , and EoO - between subgroups of protected attributes across the Adult, COMPAS and German Credit datasets for the finetuned GPT-3.5 models using three different approaches: finetuning on the entire training data (GPT-3.5 FT), oversampled data (GPT-3.5 FT OS), and undersampled data (GPT-3.5 FT US). While finetuning GPT-3.5 on task datasets (GPT-3.5 FT) can mitigate inherent social biases to some extent, the application of data resampling techniques (GPT-3.5 FT OS and GPT-3.5 FT US) during the finetuning process does not consistently yield similar results for LLMs compared to the typical mitigation observed in traditional machine learning models.

Limitations and Future Work

The authors acknowledge two primary limitations:

  1. Model Scope: The study focuses exclusively on GPT-3.5. Other state-of-the-art models (like GPT-4, Claude, or open-source alternatives like Llama) might exhibit different bias profiles.
  2. Prompt Dependency: The experiments rely on a single serialization template style. The impact of advanced prompting techniques, such as Chain-of-Thought (CoT), remains unexplored.

Future work will expand the evaluation to a broader suite of LLMs and investigate how diverse prompting strategies alter bias propagation in tabular domains.

Summary updated

Abstract

Recent literature has suggested the potential of using large language models (LLMs) to make classifications for tabular tasks. However, LLMs have been shown to exhibit harmful social biases that reflect the stereotypes and inequalities present in society. To this end, as well as the widespread use of tabular data in many high-stake applications, it is important to explore the following questions: what sources of information do LLMs draw upon when making classifications for tabular tasks; whether and to what extent are LLM classifications for tabular data influenced by social biases and stereotypes; and what are the consequential implications for fairness? Through a series of experiments, we delve into these questions and show that LLMs tend to inherit social biases from their training data which significantly impact their fairness in tabular classification tasks. Furthermore, our investigations show that in the context of bias mitigation, though in-context learning and finetuning have a moderate effect, the fairness metric gap between different subgroups is still larger than that in traditional machine learning models, such as Random Forest and shallow Neural Networks. This observation emphasizes that the social biases are inherent within the LLMs themselves and inherited from their pretraining corpus, not only from the downstream task datasets. Besides, we demonstrate that label-flipping of in-context examples can significantly reduce biases, further highlighting the presence of inherent bias within LLMs.
BibTeX
@inproceedings{liu2024confronting,
  author = {Yanchen Liu and Srishti Gautam and Jiaqi Ma and Himabindu Lakkaraju},
  title = {{Confronting LLMs with Traditional ML: Rethinking the Fairness of Large Language Models in Tabular Classifications}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-long.198},
}
Figure