From Structured Prompts to Open Narratives: Measuring Gender Bias in LLMs Through Open-Ended Storytelling

Evan Chen, Run-Jun Zhan, Yan-Bai Lin, Hung-Hsuan Chen

Introduction

Recent advancements in Large Language Model (LLM) safety alignment have aimed to mitigate harmful stereotypes. However, this research uncovers a "gender bias paradox": while alignment has led models to significantly overrepresent female characters across nearly all occupations, the relative distribution of these characters continues to mirror human stereotypes rather than real-world labor statistics. In effect, models have traded a historical male-centric bias for an artificial female-centric skew without decoupling occupations from traditional gender roles.

The Challenge of Measuring Bias

Quantifying bias in modern LLMs is increasingly difficult because models are trained to "behave" when they detect they are being evaluated. Traditional benchmarks—such as choosing a job candidate from a list or resolving ambiguous pronouns—are often "gameable." If a model recognizes a prompt as a fairness test, it may provide a socially acceptable answer that masks its underlying training distribution.

To address this, the authors utilize free-form storytelling. By asking models to simply "write a story about a [job title]," they bypass explicit safety filters and surface authentic, embedded biases within a less constrained narrative environment.

Experimental Framework

The researchers developed a four-stage evaluation framework to probe these narrative biases:

  1. Occupation Selection: A curated list of 106 occupations was created by merging U.S. labor data with psychological studies on human gender perceptions.
  2. Open-Ended Prompting: Ten prominent LLMs (including GPT-4o, Gemini 2.0, and Llama 3.2) were prompted to generate story openings for each occupation. To ensure statistical significance, 75 independent sessions were conducted per occupation for each model.
  3. Gender Classification: To avoid ambiguity in first-person narratives, the researchers extracted character names and used the nomquamgender package to infer gender, achieving a manual verification accuracy of 98.77%.
  4. Bias Analysis: Generated gender ratios were compared against two benchmarks: real-world employment data (U.S. BLS) and human stereotype ratings (GSR).

Benchmarks and Data

The study relies on three primary data sources:

  • U.S. Bureau of Labor Statistics (BLS) 2023: Representing the demographic reality of the current workforce.
  • Gender Stereotype Rating (GSR): A dataset measuring how strongly humans associate specific jobs with a particular gender on a scale of 1 to 7.
  • LLM-Generated Corpus: A new dataset of story openings across 106 occupations generated by the 10 state-of-the-art models.

The Gender Bias Paradox

The analysis reveals two conflicting trends in model behavior:

The Alignment Overcorrection: Modern LLMs exhibit a pervasive overrepresentation of females. In the generated stories, 35 out of 106 occupations were "female-dominated" (over 80% female), while only 5 were "male-dominated." This shift is visually evident when comparing modern models to older versions like GPT-2 XL, which was 82% male-biased. Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF) appear to have created a new, artificial distributional skew.

Male proportion across models compared to U.S. BLS and GSR benchmarks.

The Paradox of Relative Bias: Despite generating more women overall, the ranking of occupations by gender still follows human stereotypes. For example, while a model might make 70% of its "Engineers" female (exceeding the real-world 17%), it will still make 95% of its "Nurses" female. The stereotypical "gap" between occupations remains intact, suggesting that alignment has shifted the baseline without addressing the underlying associations.

Quantitative Findings

The results across all 10 models show a consistent median male proportion significantly below the U.S. labor force's actual male proportion of approximately 47%.

Kendall’s Tau and Averaged Deviations for LLM outputs against BLS and GSR.

As shown in the table above, the researchers used Kendall’s Tau to measure correlation. They found that LLM outputs align much more closely with human perceptions of stereotypes (GSR) than with real-world data (BLS). This pattern holds across different model families (OpenAI, Google, Meta), suggesting a common industry-wide trend in fairness alignment strategies.

Discussion and Future Directions

While the study provides a robust framework for measuring narrative bias, it notes several limitations:

  • Binary Gender: The focus remains on male and female cues, as non-binary representation is difficult to capture with current automated tools.
  • Western-Centricity: Benchmarks are based on U.S. data; gender associations vary significantly across different cultures and languages.
  • Name-Based Inference: While accurate, this method can be culturally biased or fail with gender-neutral names.

Future research should expand to include intersectional biases (race, age) and explore methods to achieve demographic balance without creating new, artificial skews that misrepresent the workforce.

Summary updated

BibTeX
@inproceedings{chen2025from,
  author = {Evan Chen and Run-Jun Zhan and Yan-Bai Lin and Hung-Hsuan Chen},
  title = {{From Structured Prompts to Open Narratives: Measuring Gender Bias in LLMs Through Open-Ended Storytelling}},
  booktitle = {ACM International Conference on Information and Knowledge Management},
  year = {2025},
  doi = {10.1145/3746252.3760969},
}
Figure