TL;DR
This paper investigates gender and occupational biases in large language models (LLMs) by analyzing their free-form text completions. The authors found that even instruction-tuned models still reflect societal biases, particularly in Italian, and struggle with gender-neutral language. This work highlights the need for more robust debiasing strategies beyond simple alignment.
What Problem Does This Paper Address?
As LLMs become increasingly integrated into sensitive applications like human resources, understanding and mitigating their inherent biases is crucial. Specifically, this paper focuses on gender and occupational biases, which can lead to unfair decision-making in hiring and other critical areas. While previous research has often relied on constrained tasks or specific prompts (like using first names), this study aims to uncover more implicit biases by analyzing how LLMs complete open-ended sentences related to occupations, considering both English and Italian to account for linguistic differences in gender marking.
Methods
The researchers developed a comprehensive evaluation framework involving three main components:
- Dataset Creation: They compiled a database of occupational titles in English and Italian, drawing from the International Standard Classification of Occupations (ISCO-08). For Italian, they specifically included masculine, feminine, and gender-neutral forms of occupational titles to capture language-specific nuances.
- Prompt Engineering: They crafted a large set of free-form prompts for each language and model, using verbs like "to be," "to be considered," and "to aspire to" to elicit completions related to an individual's identity, actions, and desires.
- Linguistic Analysis: They employed a multi-faceted approach to analyze the generated text completions:
- Human Analysis: Two annotators manually assessed over 4,500 texts across four dimensions: gender assignment, subject misinterpretation, topic focus (occupation vs. personal details), and the presence of agentic vs. communal language.
- Embedding Analysis: They used a clustering framework on sentence embeddings to identify semantic patterns and biases related to gender and occupation. This involved representing sentences as vectors and analyzing how completions clustered, particularly when the prompt was removed to isolate completion-specific biases.
- Lexical Analysis: They extracted the most informative tokens for each gender representation to identify specific words or phrases that reveal underlying biases.
Datasets
- Occupational Titles Database: Derived from the International Standard Classification of Occupations (ISCO-08), this database formed the basis for generating prompts. It includes 436 occupations in English and 813 in Italian, with Italian titles also represented in masculine, feminine, and gender-neutral forms.
- Generated Text Completions: The study generated approximately 4,062 texts for English and 18,664 for Italian across nine different LLMs. A subset of 4,500 of these texts was manually annotated.
Key Insights and Ideas
- Alignment Isn't a Silver Bullet: Instruction-tuned models, designed to be more helpful and less biased, still exhibit significant gender and occupational biases. Alignment might mask biases rather than truly eliminate them.
- Language Matters: Italian, a grammatically gendered language, presents unique challenges for LLMs compared to English. Models struggle more with gender-neutral expressions and exhibit different bias patterns in Italian.
- Implicit Biases Persist: By analyzing free-form completions, the study uncovered implicit associations between genders and occupations that might be missed in more constrained evaluation settings.
- Masculine as the Default: Across models and languages, there's a tendency to misgender feminine or neutral prompts as masculine, suggesting a strong underlying masculine default.
- Stereotypes Linger: Feminine gender representations are more frequently associated with communal language (warmth, morality) and personal details (family, appearance), while masculine representations lean towards agentic language (competence, ambition).
Key Results
- Misgendering: While masculine prompts were rarely misgendered, feminine and neutral prompts were misgendered significantly more often, especially in Italian (36.6% of completions). The assigned gender was predominantly masculine.
- Subject Misinterpretation: Italian prompts, particularly those using gender-neutral terms, were more prone to misinterpretation (e.g., as abstract entities or objects) compared to English.
- Topic Focus: Models often steered away from the occupation itself, focusing on personal identity or storytelling, particularly with feminine gender representations.
- Agency vs. Communion: Feminine representations were strongly associated with communal language in English. While instruction tuning showed some promise in increasing agency for feminine roles, it sometimes led to unintended increases in communal language as well.
- Embedding Analysis: The analysis of sentence embeddings revealed that while completions clustered well by occupation when the prompt was included, removing the prompt led to completions clustering more by gender, indicating gender-based biases in the generated text.
- Lexical Analysis: The word "woman" frequently appeared as a top token for feminine representations across models. For masculine representations, terms like "person" or "father" were common, suggesting a bias where "man" is the implicit norm.
Limitations and Future Work
- Limited Gender Identities: The study focused on masculine, feminine, and neutral gender representations, acknowledging that other gender identities (e.g., expressed through neopronouns) were not considered.
- Language Scope: While comparing English and Italian provides valuable insights, extending the analysis to more languages, especially low-resource ones, is crucial.
- Embedding Model Dependency: The embedding analysis was based on a single embedding model, which could influence results. However, the authors argue their approach of using both prompt and no-prompt embeddings mitigates this.
- Interplay of Biases: The study acknowledges the complex interplay between gender and occupational bias but notes that a granular analysis of all dimensions was beyond the scope of this paper.
- Future Work: The authors suggest further research into the nuances of gender-neutral terminology in Italian, exploring other gender identities, and conducting more in-depth analyses of specific model behaviors and the interaction between different types of biases.
Summary updated