Investigating the Fairness of Large Language Models for Predictions on Tabular Data

Yanchen Liu, Srishti Gautam, Jiaqi Ma, Himabindu LakkarajuSubmission to ICLR 2024 · Withdrawn

TL;DR

Applying Large Language Models (LLMs) to tabular data introduces severe fairness disparities inherited directly from pre-training text corpora—often producing larger demographic gaps than standard machine learning models. While in-context learning and fine-tuning moderately reduce these gaps, traditional data-level mitigations like resampling fail on LLMs, highlighting the need for bias interventions tailored to language models in high-stakes tabular domains.

What Problem Does This Paper Address?

Recent research has proposed serializing tabular datasets into natural language strings so LLMs can perform classification tasks (e.g., loan approval or recidivism prediction) with few or zero training examples. However, tabular data predominantly powers safety-critical, high-stakes decisions where algorithmic fairness is legally and ethically scrutinized.

While fairness in traditional tabular ML (e.g., tree ensembles, neural nets) has been widely studied, the fairness properties of LLMs on tabular tasks remain largely unexplored. LLMs are known to absorb societal stereotypes from their massive web text pre-training. This paper investigates:

  1. Do LLMs rely on social biases when predicting on tabular data?
  2. How does LLM fairness compare to standard tabular baselines like Random Forests and Neural Networks?
  3. Can standard mitigation strategies (in-context demonstrations, fine-tuning, data resampling, or label flipping) neutralize these biases?

Methods

The authors evaluate GPT-3.5 (gpt-3.5-turbo) alongside two tabular baselines: a Random Forest (RF) (100 trees) and a shallow 3-layer Neural Network (NN).

  • Serialization: Tabular feature-value pairs are converted into natural language strings (e.g., workclass: Private, hours per week: 40, sex: Female...) preceded by task instructions.
  • Prompting & Training Regimes:
  • Zero-shot Prompting: Direct prediction without task examples.
  • Few-shot In-Context Learning (ICL): Providing 50 randomly sampled training instances in the prompt.
  • Label-Flipped ICL: Inverting the labels of the 50 in-context demonstration examples to probe how LLMs balance semantic priors vs. explicit in-context label mappings.
  • Fine-Tuning: Full fine-tuning via OpenAI’s API.
  • Resampling: Training on datasets modified via minority oversampling or majority undersampling.
  • Fairness Metrics: Accuracy, F1 score, Statistical Parity Difference (SPD), and Equal Opportunity Difference (EOD) across sensitive demographic subgroups.

Datasets

The authors use three standard fairness benchmarks (adapted to focus on binary classification and specific protected attributes):

  • Adult Income: Predict whether income exceeds \$50K/year (Protected attribute: Sex [female as disadvantaged group]).
  • German Credit: Classify credit risk as good or bad (Protected attribute: Sex [female as disadvantaged group]).
  • COMPAS Recidivism: Predict two-year criminal recidivism (Protected attribute: Race [African-American as disadvantaged group]).

No new datasets were introduced.

Key Insights and Ideas

  • Biases are Inherent, Not Just Dataset-Driven: In zero-shot settings, GPT-3.5 exhibits massive fairness disparities—frequently wider than those observed in classical models trained directly on the biased data. This demonstrates that the model imports social stereotypes directly from its pre-training corpus.
  • Tabular ICL Depends Heavily on Labels: Prior NLP research suggested that LLMs mostly rely on semantic priors and that in-context example labels matter very little. In tabular classification, however, flipping demonstration labels drastically shifted model behavior, showing that LLMs actively attend to input-label mappings when semantic tabular priors are weaker.
  • Label-Flipping Neutralizes Bias (at a Cost): Reversing in-context demonstration labels effectively countered the model's pre-training biases and slashed fairness gaps, though at the expense of overall predictive accuracy.
  • Standard ML Debiasing Fails on LLMs: While class/subgroup resampling (undersampling/oversampling) reliably improves fairness in Random Forests and Neural Networks, it produces inconsistent and often negligible effects when fine-tuning GPT-3.5.

Key Results

  • Zero-Shot Disparity: On the Adult dataset, zero-shot GPT-3.5 showed an Equal Opportunity Difference (EOD) of $-0.483$ between male and female applicants—drastically worse than the baseline NN ($-0.123$) and RF ($-0.195$).
  • Mitigation via Few-Shot & Label Flipping:
  • Regular 50-shot prompting reduced the Adult EOD gap to $-0.271$.
  • Label-flipped 50-shot prompting nearly eliminated the gap, reducing EOD disparity to $+0.037$ (though accuracy dropped from $\approx 89.9\%$ to $68.2\%$ for females).
  • On COMPAS, label flipping reduced the EOD gap from $-0.334$ (zero-shot) down to $+0.077$.
  • Failure on Uninformative Prompts: On German Credit, zero-shot GPT-3.5 defaulted to predicting "good credit" for virtually all profiles (achieving near 50% accuracy, comparable to random guessing), indicating that ambiguous feature names prevent zero-shot LLM reasoning without richer feature descriptions.
  • Fine-Tuning: Fine-tuning GPT-3.5 brought the Adult EOD difference down to $-0.137$, matching standard ML levels, but data resampling did not consistently improve these numbers further.

Limitations and Future Work

  • Model Scope: Experiments were restricted to a single closed-source model family (gpt-3.5-turbo); open-weights models (e.g., Llama) and larger frontier models (e.g., GPT-4) were not evaluated.
  • Serialization Simplicity: The serialization scheme used raw column headers. As shown in the German Credit experiment, domain-specific abbreviations and vague headers caused model collapse in zero-shot settings; descriptive prompts could alter fairness profiles.
  • Practical Debiasing Trade-offs: Label-flipping served as an empirical diagnostic rather than a viable production strategy, as it drastically damaged classification accuracy. Designing prompt-based or parameter-efficient debiasing methods that preserve tabular utility remains an open challenge.

Summary updated

Abstract

Recent literature has suggested the potential of using large language models (LLMs) to make predictions for tabular tasks. However, LLMs have been shown to exhibit harmful social biases that reflect the stereotypes and inequalities present in the society. To this end, as well as the widespread use of tabular data in many high-stake applications, it is imperative to explore the following questions: what sources of information do LLMs draw upon when making predictions for tabular tasks; whether and to what extent are LLM predictions for tabular tasks influenced by social biases and stereotypes; and what are the consequential implications for fairness? Through a series of experiments, we delve into these questions and show that LLMs tend to inherit social biases from their training data which significantly impact their fairness in tabular prediction tasks. Furthermore, our investigations show that in the context of bias mitigation, though in-context learning and fine-tuning have a moderate effect, the fairness metric gap between different subgroups is still larger than that in traditional machine learning models, such as Random Forest and shallow Neural Networks. This observation emphasizes that the social biases are inherent within the LLMs themselves and inherited from their pre-training corpus, not only from the downstream task datasets. Besides, we demonstrate that label-flipping of in-context examples can significantly reduce biases, further highlighting the presence of inherent bias within LLMs.
BibTeX
@inproceedings{liu2024investigating,
  author = {Yanchen Liu and Srishti Gautam and Jiaqi Ma and Himabindu Lakkaraju},
  title = {{Investigating the Fairness of Large Language Models for Predictions on Tabular Data}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-long.198},
}
Figure