TL;DR
This paper quantifies how gender bias has evolved in biomedical research over the past 60 years by analyzing word embeddings derived from PubMed articles. They found that while some traditional gender stereotypes have decreased, biases in medical conditions remain stubbornly persistent, highlighting a critical area for improvement in scientific literature.
What Problem Does This Paper Address?
Gender bias in biomedical research can lead to significant health disparities. When research funding, study populations, and resulting guidance disproportionately favor one gender, the health needs of other groups may be overlooked or inadequately addressed. While previous work has identified bias in various AI applications and even in medical diagnosis, there's a need to understand how this bias has manifested and changed over time specifically within the vast corpus of biomedical literature. This paper aims to fill that gap by quantifying temporal trends in gender bias embedded within the language of scientific research.
Methods
The researchers employed word embeddings, which are numerical representations of words that capture their contextual relationships. They trained decade-specific word embeddings using the Skip-Gram model on titles and abstracts from PubMed articles spanning 1960 to 2020. To measure gender bias, they utilized three established computational techniques:
- Word Embedding Association Test (WEAT): This method quantifies bias by measuring the association between sets of target words (e.g., "career" vs. "family") and attribute words (e.g., "male" vs. "female").
- Embedding Coherence Test (ECT): ECT measures bias by assessing the correlation between how male and female attribute words relate to a single set of target words (e.g., occupations or mental health disorders). This allows for analysis of bias in individual word categories.
- Relational Inner Product Association (RIPA): RIPA is used to identify the most biased individual words by calculating the dot product between a word's embedding and a derived "gender vector."
Datasets
The primary dataset used was a collection of PubMed-indexed titles and abstracts from 1960 to 2020. For evaluating the quality of their trained word embeddings, they used the UMLS-Sim dataset, which contains medical concept pairs rated for similarity by medical experts. They also utilized word lists for occupations from prior research and mental health-related terms derived from the Diagnostic and Statistical Manual of Mental Disorders (DSM-5).
Key Insights and Ideas
- Persistent Bias in Medical Conditions: Perhaps the most striking finding is that while some traditional gender stereotypes (like "strong vs. weak") have shown a decreasing bias over time, biases related to specific medical conditions remain as strong today as they were 60 years ago. This suggests a deep-seated issue in how these conditions are discussed and understood in research.
- Shifting Occupational Bias: The bias associated with occupations has changed significantly over time. While historically, certain professions were strongly gendered in research literature, this bias has seen substantial shifts, indicating a potential evolution in how gender roles are represented in scientific discourse.
- Embedding Quality vs. Bias: The study observed an interesting correlation: as the quality of word embeddings (measured by UMLS-Sim) improved over the decades, the overall bias in the embeddings tended to decrease. However, this trend reversed in the 2000s and 2010s, possibly due to the sheer volume and diversity of topics covered in PubMed, leading to embeddings that struggle to capture single, clear meanings.
- "Hysteria" as a Lingering Bias: The term "hysteria," despite its historical context and decline in clinical use, remains heavily biased in recent decades, highlighting how outdated concepts can persist in scientific language.
Key Results
- WEAT Results: Bias scores for "Career vs. Family" and "Strong vs. Weak" showed a general decrease over time. "Math vs. Art" bias remained relatively static, with female words being more associated with math. "Intelligence vs. Appearance" showed little bias until the 1990s, after which it significantly increased.
- ECT Results: Occupations showed a notable decrease in bias from the 1960s and 1970s to the 1990s. Mental disorders, however, exhibited relatively little change in bias over the same period, with scores consistently high.
- RIPA Results: Analysis of individual words revealed that addictions (e.g., "caffeine," "cannabis") are consistently male-associated, while disorders related to appearance (e.g., "dysmorphic," "anorexia") are female-associated. The term "hysteria" was found to be heavily biased towards females in the 2010s.
Limitations and Future Work
The authors acknowledge that their study focuses solely on binary gender (Male vs. Female) and that these findings may not generalize to other gender identities. They also note that the specific word lists used for WEAT are manually curated, and bias measurements can be sensitive to these choices.
Future work suggested includes:
- Causal Studies: Investigating why certain biases exist in word embeddings, rather than just quantifying them. For example, is "hysteria" biased because of current research practices or historical usage?
- Dynamic Embeddings: Exploring dynamic word embeddings that capture changes in perception over time, as opposed to static, decade-specific embeddings.
- Beyond Binary Gender: Developing methods to measure bias across a wider spectrum of gender identities, which is challenging due to the limited number of representative words. This could involve leveraging contextual word embeddings.
- Broader Domains: Applying the analysis of historical bias to other domains beyond occupations and mental disorders.
Summary updated