Causally Testing Gender Bias in LLMs: A Case Study on Occupational Bias

Yuen Chen, Vethavikashini Chithrra Raghuram, Justus Mattern, Rada Mihalcea, Zhijing Jin

TL;DR

Existing benchmarks for measuring LLM bias often suffer from correlation-causation confusion, where prompt structures confound the actual bias being measured. This paper introduces a formal causal inference framework using do-interventions to isolate true occupational gender bias, alongside a new benchmark called OCCUGENDER. Evaluating ten state-of-the-art open-source LLMs reveals persistent, asymmetric gender stereotypes and shows that instruction tuning can surprisingly amplify certain biases.

What Problem Does This Paper Address?

While it is widely acknowledged that Large Language Models (LLMs) inherit human-like biases from their training data, accurately quantifying this bias is notoriously difficult. Existing benchmarks (such as StereoSet or CrowS-Pairs) rely on purely correlational metrics. This introduces a major issue: confounding.

For example, if a prompt template is inadvertently structured around a specific gender or grammatical voice, the model's prediction might be driven more by the template mechanics than the underlying stereotype.

Furthermore, the authors identify several other flaws in current benchmarks:

  1. Subjective labeling: Relying on crowdsourced human judgments introduces high variance.
  2. Massive prediction spaces: Asking models to generate open-ended text to fill in stereotypes introduces significant measurement noise.
  3. Demographic exclusion: Most benchmarks completely ignore non-binary genders.

Methods

To resolve these issues, the authors ground bias measurement in Judea Pearl’s causal inference framework. They model a prompt as a combination of a stereotype ($S$) and a template ($T$). To isolate the true causal effect of a stereotype on the model's gender prediction ($G$), they use a do-intervention ($do(S=s)$).

By generating templates and stereotypes completely independently, they break the spurious confounding path ($S \leftarrow C \rightarrow T \rightarrow G$) typically introduced by prompt designers.

Causal graph comparison showing how OCCUGENDER eliminates template confounding by breaking the path between the confounder and the stereotype/template selection.

Because it is impossible to test every single template in existence, the authors use ChatGPT-4o to generate a diverse set of 50 templates (25 explicit, 25 implicit). They then approximate the true causal effect by marginalizing (averaging) the model's predictions across this entire template set.

Crucially, they also flip the prediction space: instead of asking the model to predict a complex stereotype given a gender, they provide the stereotype (an occupation) and force the model to predict a simple gender pronoun, drastically narrowing the prediction space and reducing noise.

Datasets

The paper introduces OCCUGENDER, a novel benchmark designed around five key desiderata: no template confounding, objective labeling, small prediction space, evaluation of both explicit and implicit biases, and inclusion of non-binary genders.

To keep the stereotype labeling strictly objective, the authors curated a list of 40 occupations directly from the 2021 U.S. Bureau of Labor Statistics:

  • 20 Female-dominated jobs (e.g., skincare specialists, kindergarten teachers, nurses)
  • 20 Male-dominated jobs (e.g., firefighters, mechanical engineers, pilots)

They evaluated ten prominent open-source models on this dataset, including Llama-2-7B, Llama-3-8B, Mistral-7B, Gemma-7B, Gemma-2-9B, and their respective instruction-tuned variants.

Key Insights and Ideas

  • Template Confounding is a Major Threat: Early testing revealed that LLMs frequently associate active grammatical voices with male pronouns. If a benchmark doesn't decouple templates from stereotypes, this linguistic quirk can easily mask or amplify measured bias.
  • Asymmetry in Bias: LLMs exhibit a much stronger, more rigid bias when associating male-dominated jobs with men than they do when associating female-dominated jobs with women.
  • Newer Models Aren't Necessarily Better: Upgrading to a newer generation within the same model family often increased stereotypical associations. For instance, Llama-3-8B predicted male pronouns for male-dominated jobs more frequently (84.4%) than Llama-2-7B did (73.5%).
  • The Double-Edged Sword of Instruction Tuning: Instruction tuning successfully introduces an awareness of non-binary identities (leading to more "they/them" predictions). However, it also frequently amplifies binary gender biases for female-dominated occupations compared to base models.

Key Results

The quantitative evaluation proved that occupational gender bias remains rampant across all tested models:

  • Male-Dominated Jobs: For explicit prompts, the ratio of male predictions ($P_m$) ranged from 73.5% to 100%. For implicit prompts, it ranged from 72.7% to 94.1%.
  • Female-Dominated Jobs: The bias was less extreme but still prominent, with female predictions ($P_f$) ranging from 54.1% to 86.9% on explicit prompts.
  • Non-Binary Inclusion: Base models almost completely ignored non-binary genders, yielding near-zero diverse predictions. Instruction-tuned models fared better; Gemma-2-9B-Instruct demonstrated the highest non-binary awareness, capturing 13.2% (explicit) and 16.7% (implicit) diverse predictions, likely due to safety-tuning protocols.

Limitations and Future Work

  • Prompt Instability: Despite reducing variance via causal marginalization, LLMs remain highly sensitive to exact phrasings, meaning some prompt-induced noise is inevitable.
  • Geocultural Bias: The benchmark relies entirely on U.S. labor statistics. Occupational gender distributions vary wildly across different cultures and eras, meaning these specific results do not represent a global ground truth.
  • Simplified Non-Binary Representation: To keep the evaluation pipeline manageable, the authors only used "they/them" as a proxy for non-binary genders, omitting neopronouns (like "ze" or "xe") because they rarely appeared in preliminary model outputs.

Summary updated

Abstract

Generated texts from large language models (LLMs) have been shown to exhibit a variety of harmful, human-like biases against various demographics. These findings motivate research efforts aiming to understand and measure such effects. This paper introduces a causal formulation for bias measurement in generative language models. Based on this theoretical foundation, we outline a list of desiderata for designing robust bias benchmarks. We then propose a benchmark called OccuGender, with a bias-measuring procedure to investigate occupational gender bias. We test several state-of-the-art open-source LLMs on OccuGender, including Llama, Mistral, and their instruction-tuned versions. The results show that these models exhibit substantial occupational gender bias. Lastly, we discuss prompting strategies for bias mitigation and an extension of our causal formulation to illustrate the generalizability of our framework. Our code and data https://github.com/chenyuen0103/gender-bias.
BibTeX
@inproceedings{chen2025causally,
  author = {Yuen Chen and Vethavikashini Chithrra Raghuram and Justus Mattern and Rada Mihalcea and Zhijing Jin},
  title = {{Causally Testing Gender Bias in LLMs: A Case Study on Occupational Bias}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
  doi = {10.18653/v1/2025.findings-naacl.281},
}
Figure