Investigating Gender Bias in STEM Job Advertisements

Malika Dikshit, Houda Bouamor, Nizar HabashACL 2024

TL;DR

This paper introduces a novel NLP approach to detect gender bias in STEM job advertisements, moving beyond simple word counts to analyze the contextual language. By categorizing job descriptions into "agentic," "balanced," and "communal" language, the study reveals how subtle linguistic choices can perpetuate gender stereotypes and discourage potential applicants, particularly women, from applying to STEM roles.

What Problem Does This Paper Address?

Gender inequality remains a persistent issue in academia, especially within STEM fields. While initiatives aim to promote diversity, the language used in job advertisements can inadvertently reinforce gender stereotypes. Previous studies often relied on identifying individual masculine- or feminine-coded words, which can be simplistic and miss the nuances of how language is used in context. This can lead to job descriptions that, while seemingly neutral, may subtly favor candidates with agentic traits (associated with men) over communal traits (associated with women), thus discouraging a diverse applicant pool. This paper aims to provide a more sophisticated method for detecting and understanding this bias.

Methods

The authors developed a novel methodology that combines Natural Language Processing (NLP) techniques with a nuanced categorization of language.

  • Sentence Embeddings: They used the Sentence-Transformers framework (specifically all-mpnet-base-v2) to generate dense vector representations (embeddings) for each sentence in the job descriptions. These embeddings capture the semantic meaning of sentences.
  • Contrastive Learning: The Sentence-Transformer model was fine-tuned using a contrastive learning objective. This process helps the model learn to distinguish between sentences with similar meanings (high similarity) and those with different meanings (low similarity), allowing it to capture subtle linguistic cues.
  • Clustering: K-means clustering with k=3 was applied to these sentence embeddings to group sentences into three categories:
  • Agentic: Emphasizing traits like assertiveness, ambition, and self-reliance.
  • Communal: Highlighting collaboration, empathy, and inclusivity.
  • Balanced: Combining aspects of both or lacking strong alignment with either.
  • Job-Level Labeling: Two techniques were used to label entire job descriptions:
  • T1 (Dictionary Look-up): Counting masculine- and feminine-coded words based on existing lexicons.
  • T2 (Embedding-based): Averaging sentence embeddings for a job description and then using K-means clustering to assign a label.

Datasets

  • Newly Introduced Dataset: The authors created a novel dataset of 6,031 academic STEM job descriptions. This dataset was compiled semi-automatically using web-scraping techniques from various academic job databases, including HigherEdJobs, TimesHigherEducation, Jobs.ac.uk, AcademicJobsOnline, and The Chronicle of Higher Education. The job ads were collected from a global scope.

Key Insights and Ideas

  • Beyond Word Counts: The study emphasizes that simply counting gender-coded words is insufficient. The context in which words are used significantly impacts whether they convey bias. For example, "competitive" can describe a challenging work environment (agentic) or market compensation (neutral).
  • Agentic Dominance: Agentic language appears to be more prevalent in STEM job descriptions, particularly in sections describing ideal candidates. This can create an environment that implicitly favors traits traditionally associated with male applicants.
  • Balanced Language as the Norm: Balanced sentences, which combine agentic and communal traits or are neutral, are the most common type found in job ads. This suggests an evolving trend towards more inclusive language, but it doesn't negate the impact of agentic language.
  • Shifting Language in Job Ads: The analysis of job ads divided into sections (top/bottom halves, or thirds) reveals a potential shift in language. While the top often contains balanced language, the bottom sections can lean more towards agentic or communal descriptions, with agentic language often appearing in the latter parts.
  • Critique of Binary Gender Coding: The paper explicitly critiques the traditional binary approach (masculine/feminine coded words) used by prior research, arguing it can reinforce harmful stereotypes. The proposed agentic/balanced/communal framework offers a more nuanced and inclusive perspective.

Key Results

  • Dataset Size: A dataset of 6,031 academic STEM job descriptions was created.
  • Clustering Performance: The clustering model showed good separation (CHI score of 787.02) but also some overlap (DBS score of 4.03), indicating discernible patterns but also potential ambiguities.
  • Sentence Distribution: Communal sentences were the least prevalent (<20%), agentic sentences ranged from 20-40%, and balanced sentences were the most common (40-60%).
  • Job-Level Labeling (T1): Agentic-labeled job ads were most prevalent, followed by communal and then balanced.
  • Job-Level Labeling (T2): Balanced-labeled job ads were the most prevalent, suggesting that many job descriptions combine agentic and communal traits or are neutral.
  • Sectional Analysis: Agentic language was found to be more prevalent in the bottom part of job descriptions compared to the top.

Limitations and Future Work

  • Subjectivity of Bias: Identifying and quantifying gender bias is inherently complex and subjective, varying among readers.
  • Dataset Snapshot: The dataset represents job descriptions from a specific time, which might not fully capture evolving trends.
  • Nuance in Embeddings: Averaging sentence embeddings might overlook specific contextual nuances, potentially leading to misclassifications.
  • Cluster Ambiguity: The distinctions between agentic, communal, and balanced language are not always clear-cut, leading to potential overlaps and misclassifications.

Future Work Suggestions:

  • Conduct qualitative research to survey participants' perceptions of gender bias in job ads.
  • Investigate the impact of language transitions within job descriptions on candidate perception, especially considering readers' tendency to lose focus towards the end of documents.
  • Explore the role of cultural factors in language clustering and bias.
  • Expand the analysis to include other minority dimensions beyond agentic and communal traits.

Summary updated

BibTeX
@inproceedings{dikshit2024investigating,
  author = {Malika Dikshit and Houda Bouamor and Nizar Habash},
  title = {{Investigating Gender Bias in STEM Job Advertisements}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure