TL;DR
This paper argues that diversity in NLP is heavily concentrated in "fairness" research, leaving core NLP areas with significantly less representation from women and researchers from non-Western/non-North American backgrounds. The authors propose actionable recommendations to address these systemic issues and foster broader inclusivity across the entire field.
What Problem Does This Paper Address?
Natural Language Processing (NLP), like many areas of computer science, has historically been dominated by a "WEIRD" demographic (Western, Educated, Industrialized, Rich, and Developed). While there's been a push for diversity and inclusion in NLP, this paper highlights a critical issue: this progress is disproportionately concentrated in "fairness" and "ethics" related subfields. This leaves core NLP areas like machine translation, sentiment analysis, and question answering with significantly lower representation from women and researchers from diverse geographic backgrounds. The authors posit that this imbalance isn't just a matter of self-selection but is driven by systemic incentives, biases, and barriers that disenfranchise researchers in non-fairness fields or push them towards fairness-related work.
Methods
The authors conducted a novel empirical study to quantify diversity across different NLP subfields. Their methodology involved:
- Keyword-based Researcher Identification: They used keywords identified from existing NLP research (e.g., "Language Models," "Machine Translation," "Bias," "Fairness") to search the ACL Anthology.
- Author Profiling: For each keyword, they identified the top 50 prolific and active authors. They then determined the gender of these authors by examining their ACL profiles, personal websites, and institutional affiliations, using traditional gendered pronouns as indicators. They explicitly avoided using names as a proxy for gender due to methodological and social concerns.
- Geographic Affiliation Mapping: They recorded the continent of each author's current affiliated institution, using UN geoscheme categories. This was chosen over birthplace or nationality to reflect current research engagement.
Datasets
The study did not introduce new datasets. Instead, it utilized existing data from:
- ACL Anthology: This was the primary source for identifying research keywords and authors.
- ACL Membership Statistics: Used for comparison with broader field-level diversity data.
- Publicly available author profiles: From institutional websites, personal pages, and professional platforms (like LinkedIn/X.com).
Key Insights and Ideas
- "Fairness" as a Diversity Hub: The most striking insight is that "fairness-related" NLP research exhibits significantly higher gender diversity (34.7% women) compared to "non-fairness related" fields (17.8% women).
- Geographic Disparities Mirror Gender Gaps: Similar to gender, North America is overrepresented in fairness research, while Asia is underrepresented, mirroring the trends seen in gender diversity.
- Self-Selection vs. Systemic Issues: While it's understandable that individuals impacted by a lack of diversity might gravitate towards fairness research, the paper strongly argues that systemic issues (cultural climates, funding, hiring practices, lack of role models) are actively contributing to this divide.
- Feedback Loops Reinforce Imbalance: The paper suggests that a lack of diverse role models in core NLP areas creates a self-perpetuating cycle, discouraging underrepresented groups from entering these fields.
- Linguistic and Geographic Barriers Push Researchers: The dominance of English and a limited number of languages in NLP datasets and models can de facto push researchers from non-English speaking backgrounds into areas like linguistic diversity research, rather than core NLP tasks.
Key Results
- Gender Diversity:
- Fairness-related fields: Average 34.7% women.
- Non-fairness related fields: Average 17.8% women.
- This is a stark difference, with fairness fields having nearly double the representation of women.
- Geographic Diversity:
- Fairness-related fields: North America (48%), Europe (39.3%), Asia (10%).
- Non-fairness related fields: North America (40.3%), Europe (33%), Asia (26.5%).
- While North America is more represented in fairness, Asia is significantly less represented in fairness-related fields compared to core NLP.
- Notably, all researchers from Oceania and one of the two from Africa in the dataset appeared in fairness-related keywords.
Limitations and Future Work
The authors acknowledge several limitations:
- Gender Identification Accuracy: Reliance on publicly available profiles might lead to misclassifications, especially for authors using non-traditional pronouns or presenting differently in professional vs. personal life. The ACL Anthology might also not adequately link trans authors who have published under different names.
- Geographic Data Nuance: Current institutional affiliations may not fully represent global distribution, particularly for researchers in less-documented regions. The continental categorization also simplifies regional diversity.
- Aggregate Data: Presenting data in aggregate reduces the risk of misgendering individual authors but might obscure specific nuances.
Future work should focus on:
- Improving Diversity Statistics: The ACL needs more transparent and granular diversity reporting, crosstabulated by research field and topic.
- Breaking Down Feedback Loops: Actively promoting and featuring diverse role models in non-fairness NLP areas through keynotes, panels, and mentorship programs. Implementing practices like citation diversity statements.
- Addressing Geographical and Linguistic Barriers: Creating more open-source datasets for underrepresented languages, developing multilingual models, and prioritizing linguistic diversity in training data. Establishing fellowship programs and grants specifically for researchers from diverse linguistic and geographic backgrounds. Structural changes in publication incentives and reviewer expectations are also crucial.
Summary updated