TL;DR
This paper introduces a novel, scalable, and dynamic method for automatically detecting lexical gender in English words by analyzing dictionary definitions. This approach overcomes the limitations of static, manually compiled lexicons, offering higher coverage and adaptability for gender bias research in NLP.
What Problem Does This Paper Address?
Gender bias in language models and datasets is a significant concern, leading to allocational and representational harms. Current research often relies on manually curated lists of gendered terms (like pronouns or words implying gender) to identify and mitigate this bias. However, these lexicons are static, can become outdated quickly, have limited coverage, and are subject to the subjective judgments of annotators. This paper addresses the need for a more dynamic, scalable, and comprehensive method to identify lexical gender, enabling more robust analysis and mitigation of gender bias in NLP.
Methods
The core of the proposed method is an algorithmic approach that leverages online lexical databases (Merriam-Webster, WordNet, and Dictionary.com). Instead of relying on pre-compiled lists, the algorithm queries the definitions of target words. It identifies lexical gender by counting the occurrences of a predefined set of definitively gendered words (e.g., "man," "woman," "male," "female") within these definitions.
The algorithm works as follows:
- Definition Retrieval: Definitions for a target word are retrieved from multiple online dictionaries.
- Gendered Word Counting: The algorithm counts occurrences of specific masculine and feminine word pairs within the retrieved definitions.
- Labeling: If the count of masculine terms exceeds feminine terms, the word is labeled as masculine. Conversely, if feminine terms are more frequent, it's labeled feminine. If counts are equal or no gendered terms are found, it's labeled neutral.
- Combined Labeling: A majority vote across the different dictionaries is used to assign a final label.
Additionally, the researchers incorporated morphological heuristics to handle cases with explicit gender markers (like "-man" or "-ess") and punctuation variations.
Datasets
The paper utilizes the following datasets for evaluation:
- Gold Standard Dataset: A curated list of nouns with unambiguous lexical gender, compiled from previous research on gender bias and online resources for English learners. This dataset consists of 53 masculine, 53 feminine, and 28 neutral words.
- Wiki1000-sample: A dataset of 1,000 randomly sampled Wikipedia articles. Nouns from these articles were extracted and analyzed for lexical gender. This resulted in a dataset of 515 nouns that were manually labeled by the research team.
- Wiki1000 dataset: A larger dataset of 12,643 nouns extracted from the 1,000 Wikipedia articles, used for evaluating the algorithm's performance on a more naturalistic corpus. For this dataset, combined predicted labels were used as gold labels were not available.
Key Insights and Ideas
- Dictionary Definitions as a Rich Source: The paper demonstrates that dictionary definitions themselves contain sufficient information to infer lexical gender, moving beyond the reliance on explicit gendered terms in text.
- Scalability and Dynamism: The algorithmic approach offers significant advantages over manual lexicons by being scalable to large corpora and dynamic, adapting to evolving language use as dictionaries are updated.
- Cross-Dictionary Consistency: Combining results from multiple dictionaries (Merriam-Webster, WordNet, Dictionary.com) leads to more robust and accurate gender detection.
- Challenges with Neutral Terms: The research highlights that neutral terms can be misclassified as gendered due to societal biases reflected in dictionary definitions (e.g., "nurse" being associated with feminine due to historical gender distributions).
- Outdated Definitions and Societal Bias: The study acknowledges that even objective resources like dictionaries can contain outdated definitions or reflect historical societal biases, leading to misclassifications.
Key Results
The proposed method achieved impressive results:
- Accuracy: The combined approach reached up to 87% accuracy on the gold standard dataset and 84% accuracy on the Wiki1000-sample.
- Dictionary Performance: On the gold standard, Dictionary.com alone achieved 87% accuracy, while on the Wiki1000-sample, Merriam-Webster alone achieved 82% accuracy.
- Parameter Optimization: Through grid search, the optimal parameters were found to be
d=4(number of definitions) andt=20(number of tokens per definition), withw=5(number of gendered word pairs) yielding the best results. - Qualitative Findings: Confusion matrices revealed that the algorithm was more prone to misclassifying neutral words as gendered than vice versa. Cases of misclassification were often linked to definitions reflecting traditional gender associations or historical biases.
Limitations and Future Work
The authors acknowledge several limitations and suggest future directions:
- Dictionary Biases: Dictionaries, while aiming for objectivity, can still contain human biases and outdated definitions, leading to misclassifications.
- Grammatical Gender: The method is specifically designed for English, which lacks grammatical gender. Adapting it to languages with grammatical gender would require significant modifications.
- Low-Resource Languages: Adapting the method to low-resource languages, where comprehensive lexical resources might be scarce, presents a challenge.
- Word Sense Disambiguation (WSD): The current method does not perform WSD, meaning it treats all senses of a word equally. For words with multiple senses (e.g., "colt" as a horse vs. a pistol), this can lead to misclassification. Future work will integrate WSD into the algorithm.
- Further Heuristics: Exploring additional morphological heuristics or other linguistic features could further improve performance.
Summary updated