BAD: BiAs Detection for Large Language Models in the context of candidate screening

Nam Ho Koh, Joseph Plata, Joyce ChaiarXiv 2023 · OpenReview

TL;DR

This paper investigates the potential for Large Language Models (LLMs) like ChatGPT to perpetuate societal biases in the context of candidate screening. The authors developed novel methods to generate and analyze resumes, revealing that LLMs can exhibit stereotypical associations between demographics and job roles, mirroring existing human biases.

What Problem Does This Paper Address?

The hiring process, traditionally manual, has seen the rise of Applicant Tracking Systems (ATS) to manage large volumes of applications. However, ATS can embed human biases into their algorithms. With the increasing adoption of LLMs in HR and recruitment, there's a growing concern that these powerful models, trained on vast amounts of real-world data, might inadvertently amplify existing societal biases, leading to unfair or discriminatory outcomes in candidate screening. This paper aims to identify and quantify these social biases within LLMs specifically in the context of job application screening.

Methods

The researchers employed a two-pronged approach:

  1. Resume Generation and Analysis: They used ChatGPT to generate sample resumes. To do this, they first created datasets of names correlated with specific gender and ethnicity pairings. Then, they prompted ChatGPT to create realistic resumes for individuals with these names, ensuring all fields were populated with plausible, real-world data. The generated resumes were then analyzed to identify patterns and associations between demographic attributes (gender, ethnicity) and job titles/areas.
  2. Context Awareness Test (CAT): Inspired by existing work, they developed a custom CAT to assess LLM bias. This test presents a context (e.g., "He is an Asian man") and asks the LLM to choose the most appropriate continuation from three options: a stereotypical association, an anti-stereotypical association, or a neutral, unrelated option. By analyzing the LLM's choices, they could quantify its tendency towards stereotypes. They also introduced a "neutral selection score" to measure how often the model opts for a neutral response over a biased one.

Datasets

The study utilized several datasets:

  • Name Datasets:
    • Harvard Dataverse "Demographic aspects of first names" dataset: For ethnic breakdowns of first names.
  • FiveThirtyEight "Most Common Names" dataset: For surname data.
  • "US Likelihood of Gender by Name" dataset: To estimate the gender associated with generated names.
  • Labor Force Data:
    • "US Labor Force Statistics" dataset from the U.S. Bureau of Labor Statistics: Used as a baseline to compare generated resume data against the actual labor market distribution of genders and ethnicities across job categories.

The researchers also created a novel dataset of 240 generated resumes, evenly distributed across eight gender-ethnicity pairings, which serves as a benchmark for evaluating LLM bias in resume generation.

Key Insights and Ideas

  • LLMs Mirror Societal Biases: The study strongly suggests that LLMs, like ChatGPT, learn and can perpetuate societal biases present in their training data, particularly in sensitive areas like job screening.
  • Stereotypical Job Associations: ChatGPT demonstrated a tendency to associate certain demographic groups with specific job roles (e.g., males and API individuals with Software Engineering, females and white individuals with Marketing).
  • Nuance in Bias Measurement: The CAT, with its inclusion of a neutral option, provides a more nuanced understanding of bias compared to simply measuring stereotypical vs. anti-stereotypical choices.
  • Model Differences: GPT-4 showed more "idealistic" behavior (higher neutral selection score) than GPT-3.5-turbo, suggesting that newer or more advanced models might be less prone to overt bias, though not entirely free of it.
  • Temperature Matters: Higher "temperature" settings (more randomness) in LLM generation led to lower performance on the CAT, indicating that less deterministic outputs can sometimes be more biased.

Key Results

  • Resume Generation: ChatGPT showed statistically significant associations between estimated ethnicity and job area (chi-squared = 58.96, p-value = 1.829e-05) and between job title and estimated ethnicity (chi-squared = 72.3, p-value = 0.00031). For instance, API males were over 8 times as likely to be represented in Software Engineering in the labor force data, and 2.2 times as likely in ChatGPT outputs. White females were notably absent from Software Engineering roles in the generated data.
  • Context Awareness Test (CAT): GPT-4 generally outperformed GPT-3.5-turbo across various temperature settings, achieving higher Idealized CAT (ICAT) scores. At temperature 0, GPT-4 yielded the highest ICAT score (23.44), while GPT-3.5-turbo's best score was 15.62 (at temperature 0.7). Notably, higher temperatures (more randomness) generally led to lower ICAT scores, indicating increased bias.

Limitations and Future Work

  • Dataset Size: The generated resume dataset (240 samples) and CAT dataset (16 examples) are relatively small and may not fully represent the complexities of the wider US population.
  • Model Refusals: GPT-4 sometimes refused to answer questions that conflicted with OpenAI's ethical guidelines, requiring workarounds to obtain responses.
  • Limited Scope: The analysis primarily focused on job titles and job areas. Future work could explore other attributes like educational achievement, location, and zip codes.
  • Broader LLM Landscape: The study focused on OpenAI models. Expanding the analysis to other LLMs (e.g., Bard, Alpaca) is a key future direction.
  • College Admissions: The authors suggest applying similar bias detection methods to college admissions processes.
  • Unisex Names: Further investigation into the impact of unisex names on bias detection is warranted.

Summary updated

Abstract

Application Tracking Systems (ATS) have allowed talent managers, recruiters, and college admissions committees to process large volumes of potential candidate applications efficiently. Traditionally, this screening process was conducted manually, creating major bottlenecks due to the quantity of applications and introducing many instances of human bias. The advent of large language models (LLMs) such as ChatGPT and the potential of adopting methods to current automated application screening raises additional bias and fairness issues that must be addressed. In this project, we wish to identify and quantify the instances of social bias in ChatGPT and other OpenAI LLMs in the context of candidate screening in order to demonstrate how the use of these models could perpetuate existing biases and inequalities in the hiring process.
BibTeX
@misc{koh2023bad,
  author = {Nam Ho Koh and Joseph Plata and Joyce Chai},
  title = {{BAD: BiAs Detection for Large Language Models in the context of candidate screening}},
  year = {2023},
  eprint = {2305.10407},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.10407},
}
Figure