TL;DR
This paper re-examines a prior study claiming gender bias in dependency parsing, finding the original results irreproducible and likely due to a methodological flaw in data preprocessing. The authors demonstrate that sentence length, not gender, is the primary factor influencing parsing difficulty, challenging previous conclusions about "women's syntactic resilience."
What Problem Does This Paper Address?
Natural Language Processing (NLP) models are often trained on large datasets that reflect existing societal imbalances, such as authorship demographics. A previous study by Garimella et al. (2019) claimed that dependency parsers exhibit gender bias, performing differently on sentences written by men versus women. Specifically, they suggested that women's sentences were more "diverse" and complex, leading to parsing challenges, while men's sentences were less variable. This paper aims to rigorously re-evaluate these claims by attempting to reproduce the original findings and investigate potential underlying causes for any observed differences. The core problem is to determine if genuine gender bias exists in NLP parsing or if apparent biases are artifacts of data or methodology.
Methods
The authors largely adopt the experimental setup of Garimella et al. (2019) to ensure reproducibility. They re-run parsing experiments using the same datasets and a similar methodology. To control for potential architectural influences, they employ two types of dependency parsers: a transition-based (TB) parser and a graph-based (GB) parser. They also extend the original study by testing on both English and German data. A key aspect of their methodology is the use of statistical significance testing (Wilcoxon signed-rank test) to validate observed differences, a step they note was absent in the original work. Furthermore, they conduct an in-depth error analysis, examining factors like dependency length and distance to the root to understand why certain sentences might be harder to parse.
Datasets
The study uses two primary treebanks:
- Penn Treebank (English): The same gender-annotated subset used by Garimella et al. (2019), containing approximately 19,399 trees from male authors and 7,282 from female authors.
- TIGER 2.2 (German): A syntactically annotated German corpus. A subset was annotated with author gender by Falenska et al. (2018), resulting in 3,550 trees from female authors and 15,184 from male authors.
The authors explicitly mention that these datasets categorize gender in binary terms (female/male), acknowledging this limitation excludes non-binary individuals.
Key Insights and Ideas
- Reproducibility Failure: The most striking finding is that the original results of Garimella et al. (2019) are not reproducible. The patterns suggesting gender bias did not hold up under re-examination.
- Methodological Oversight: The paper identifies a significant methodological flaw in the original study's preprocessing. The balancing procedure, intended to equalize dataset sizes, inadvertently created longer male sentences and shorter female sentences.
- Sentence Length as the True Factor: The authors demonstrate a strong correlation between sentence length and parsing difficulty. Shorter sentences are consistently easier to parse, regardless of the model or gender of the author. This directly explains the observed "easier to parse" phenomenon for female-authored sentences in their re-analysis.
- Lack of Generalization: The claimed gender biases did not generalize to different languages (German) or parsing architectures (graph-based vs. transition-based).
- Statistical Significance is Crucial: The absence of statistical significance testing in the original study is highlighted as a major weakness, especially given the small accuracy differences and the inherent randomness of neural models.
Key Results
- English TB Results: In their re-analysis, the FEMALE model performed best on female data, and the GENERIC model excelled on both male and generic data. This contrasts with Garimella et al.'s finding that the MALE model performed best on male data. Statistical significance was mixed and not consistently in favor of the original claims.
- German Results: For the German dataset, the GENERIC model achieved the highest scores on female data, and the MALE model performed best on male and generic data (UAS/LAS). However, these differences were generally not statistically significant.
- Error Analysis: Analysis of dependency length and distance to root showed that shorter dependencies and distances to root were more common in female-authored sentences (in the BALANCED dataset), leading to easier parsing. This pattern was directly linked to the preprocessing step that altered sentence lengths.
Limitations and Future Work
- Binary Gender Classification: The study acknowledges the limitation of using only binary gender categories (male/female) due to the available datasets, thus excluding non-binary individuals and contributing to recognition bias.
- Focus on Syntactic Differences: The authors suggest future work should focus more on specific syntactic differences related to demographic variations rather than relying solely on average scores, which can be misleading. They advocate for a deeper understanding of why these differences occur, moving beyond simple bias claims.
- Limited Scope of Original Study: The original study's limitations (English only, single architecture) are reiterated, underscoring the need for broader investigations.
Summary updated