TL;DR
Applying Large Language Models (LLMs) to tabular data introduces severe fairness disparities inherited directly from pre-training text corpora—often producing larger demographic gaps than standard machine learning models. While in-context learning and fine-tuning moderately reduce these gaps, traditional data-level mitigations like resampling fail on LLMs, highlighting the need for bias interventions tailored to language models in high-stakes tabular domains.
What Problem Does This Paper Address?
Recent research has proposed serializing tabular datasets into natural language strings so LLMs can perform classification tasks (e.g., loan approval or recidivism prediction) with few or zero training examples. However, tabular data predominantly powers safety-critical, high-stakes decisions where algorithmic fairness is legally and ethically scrutinized.
While fairness in traditional tabular ML (e.g., tree ensembles, neural nets) has been widely studied, the fairness properties of LLMs on tabular tasks remain largely unexplored. LLMs are known to absorb societal stereotypes from their massive web text pre-training. This paper investigates:
- Do LLMs rely on social biases when predicting on tabular data?
- How does LLM fairness compare to standard tabular baselines like Random Forests and Neural Networks?
- Can standard mitigation strategies (in-context demonstrations, fine-tuning, data resampling, or label flipping) neutralize these biases?
Methods
The authors evaluate GPT-3.5 (gpt-3.5-turbo) alongside two tabular baselines: a Random Forest (RF) (100 trees) and a shallow 3-layer Neural Network (NN).
- Serialization: Tabular feature-value pairs are converted into natural language strings (e.g.,
workclass: Private, hours per week: 40, sex: Female...) preceded by task instructions. - Prompting & Training Regimes:
- Zero-shot Prompting: Direct prediction without task examples.
- Few-shot In-Context Learning (ICL): Providing 50 randomly sampled training instances in the prompt.
- Label-Flipped ICL: Inverting the labels of the 50 in-context demonstration examples to probe how LLMs balance semantic priors vs. explicit in-context label mappings.
- Fine-Tuning: Full fine-tuning via OpenAI’s API.
- Resampling: Training on datasets modified via minority oversampling or majority undersampling.
- Fairness Metrics: Accuracy, F1 score, Statistical Parity Difference (SPD), and Equal Opportunity Difference (EOD) across sensitive demographic subgroups.
Datasets
The authors use three standard fairness benchmarks (adapted to focus on binary classification and specific protected attributes):
- Adult Income: Predict whether income exceeds \$50K/year (Protected attribute: Sex [female as disadvantaged group]).
- German Credit: Classify credit risk as good or bad (Protected attribute: Sex [female as disadvantaged group]).
- COMPAS Recidivism: Predict two-year criminal recidivism (Protected attribute: Race [African-American as disadvantaged group]).
No new datasets were introduced.
Key Insights and Ideas
- Biases are Inherent, Not Just Dataset-Driven: In zero-shot settings, GPT-3.5 exhibits massive fairness disparities—frequently wider than those observed in classical models trained directly on the biased data. This demonstrates that the model imports social stereotypes directly from its pre-training corpus.
- Tabular ICL Depends Heavily on Labels: Prior NLP research suggested that LLMs mostly rely on semantic priors and that in-context example labels matter very little. In tabular classification, however, flipping demonstration labels drastically shifted model behavior, showing that LLMs actively attend to input-label mappings when semantic tabular priors are weaker.
- Label-Flipping Neutralizes Bias (at a Cost): Reversing in-context demonstration labels effectively countered the model's pre-training biases and slashed fairness gaps, though at the expense of overall predictive accuracy.
- Standard ML Debiasing Fails on LLMs: While class/subgroup resampling (undersampling/oversampling) reliably improves fairness in Random Forests and Neural Networks, it produces inconsistent and often negligible effects when fine-tuning GPT-3.5.
Key Results
- Zero-Shot Disparity: On the Adult dataset, zero-shot GPT-3.5 showed an Equal Opportunity Difference (EOD) of $-0.483$ between male and female applicants—drastically worse than the baseline NN ($-0.123$) and RF ($-0.195$).
- Mitigation via Few-Shot & Label Flipping:
- Regular 50-shot prompting reduced the Adult EOD gap to $-0.271$.
- Label-flipped 50-shot prompting nearly eliminated the gap, reducing EOD disparity to $+0.037$ (though accuracy dropped from $\approx 89.9\%$ to $68.2\%$ for females).
- On COMPAS, label flipping reduced the EOD gap from $-0.334$ (zero-shot) down to $+0.077$.
- Failure on Uninformative Prompts: On German Credit, zero-shot GPT-3.5 defaulted to predicting "good credit" for virtually all profiles (achieving near 50% accuracy, comparable to random guessing), indicating that ambiguous feature names prevent zero-shot LLM reasoning without richer feature descriptions.
- Fine-Tuning: Fine-tuning GPT-3.5 brought the Adult EOD difference down to $-0.137$, matching standard ML levels, but data resampling did not consistently improve these numbers further.
Limitations and Future Work
- Model Scope: Experiments were restricted to a single closed-source model family (
gpt-3.5-turbo); open-weights models (e.g., Llama) and larger frontier models (e.g., GPT-4) were not evaluated. - Serialization Simplicity: The serialization scheme used raw column headers. As shown in the German Credit experiment, domain-specific abbreviations and vague headers caused model collapse in zero-shot settings; descriptive prompts could alter fairness profiles.
- Practical Debiasing Trade-offs: Label-flipping served as an empirical diagnostic rather than a viable production strategy, as it drastically damaged classification accuracy. Designing prompt-based or parameter-efficient debiasing methods that preserve tabular utility remains an open challenge.
Summary updated