TL;DR
This paper introduces SoWinoBias, a new benchmark for detecting subtle, "second-order" gender bias in coreference resolution systems. They demonstrate that even without explicit gendered words, systems can exhibit bias, and evaluate existing debiasing methods, finding that addressing direct bias is more effective than indirect bias mitigation for this specific task.
What Problem Does This Paper Address?
Coreference resolution systems aim to identify which noun phrases in a text refer to the same entity. While explicit gender bias (e.g., "The doctor hired the secretary because he was overwhelmed" vs. "she was overwhelmed") has been studied and addressed, this paper tackles a more insidious form: latent or second-order gender bias. This bias emerges even when sentences lack explicit gendered pronouns or nouns, relying instead on societal stereotypes associated with occupations and adjectives. For instance, a system might perform better on sentences where a "nurse" is described as "beautiful" (aligning with stereotypes about women and appearance) compared to when a "doctor" is described as "beautiful." This latent bias can perpetuate harmful stereotypes by implying that certain occupations are inherently tied to specific genders or attributes, even when the language used is gender-neutral.
Methods
The authors introduce the SoWinoBias test set, designed to measure this second-order gender bias. It consists of sentences structured as "[occupation 1] (dis)liked [occupation 2] because they were [adjective]". The key is that "they" is used as a gender-neutral pronoun, and the bias is measured by comparing performance on pro-stereotypical sentences (where the adjective aligns with stereotypes about the second occupation) versus anti-stereotypical sentences (where it contradicts them).
They evaluate four existing word embedding debiasing methods:
- Hard-GloVe: Projects gender-neutral words into the nullspace of the "gender direction" to eliminate direct bias.
- GN-GloVe: A retraining method that attempts to sequester gender information to the final embedding component.
- RAN-GloVe: Aims to repel undue gender proximities among gender-neutral words.
- INLP-GloVe: Uses iterative nullspace projection to learn an orthogonal subspace for gender.
These debiasing methods are applied to static word embeddings, which are then used in a state-of-the-art coreference resolution model that also utilizes contextual embeddings (ELMo). The paper also explores the impact of data augmentation, specifically gender-swapping in the training data.
Datasets
- SoWinoBias: A newly introduced test set for measuring latent gender bias in coreference resolution. It contains 4096 pro-stereotypical and 4096 anti-stereotypical sentences.
- WinoBias: An existing test set used for comparison, which measures explicit gender bias.
- OntoNotes 5.0: Used for training and evaluating the coreference resolution system.
Key Insights and Ideas
- Latent bias is real and problematic: The paper convincingly demonstrates that coreference resolution systems can exhibit significant gender bias even when no explicit gendered words are present, relying on subtle societal stereotypes.
- Direct bias mitigation is more effective for this task: Methods focused on removing direct gender bias from word embeddings (like Hard-GloVe) showed better results on SoWinoBias than methods designed to address indirect bias (like RAN and INLP).
- Data augmentation is a powerful ally: Simple gender-swapping in the training data significantly improved debiasing performance, especially when combined with certain debiased embeddings (like GN-GloVe). This suggests that addressing bias at both the embedding and data levels can be highly effective.
- WEAT is not a perfect proxy for downstream bias: While debiasing methods might reduce associations in embedding space tests like WEAT, this doesn't always translate to reduced bias in the downstream coreference resolution task.
- Contextual embeddings matter: The coreference model uses both static and contextual embeddings. The debiasing was applied only to static embeddings, and the results suggest that this can be sufficient to mitigate latent bias, without needing to debias the contextual embeddings themselves.
Key Results
- SoWinoBias performance: Without debiasing, systems show a significant performance difference between pro-stereotypical and anti-stereotypical sentences on SoWinoBias.
- Debiasing effectiveness: Hard-GloVe, when combined with data augmentation, reduced the performance difference on SoWinoBias to 2.1% (from a higher baseline), while maintaining overall performance.
- RAN-GloVe: While RAN-GloVe showed promise on WinoBias (explicit bias) and in embedding space tests (like WEAT), it did not significantly reduce the latent bias measured by SoWinoBias, especially without data augmentation.
- INLP-GloVe: Showed good results in reducing recoverability of gender labels in embedding space, but did not effectively mitigate the latent bias in SoWinoBias.
Limitations and Future Work
- Binary gender focus: The current study is limited to binary gender. Future work could extend the SoWinoBias test set to include other demographic biases.
- Stereotype dependence: The construction of SoWinoBias relies on knowledge of expected relationships and stereotypes, which may vary across cultures and languages. Interdisciplinary work with social sciences is encouraged.
- Complexity of "they": The paper acknowledges the potential semantic ambiguity of the pronoun "they," although they intended it as a strictly gender-neutral pronoun.
- Future directions: The authors suggest exploring bias assessment tasks that specifically probe indirect bias and non-linear recoverability. They also propose extending SoWinoBias to more complex, "in the wild" sentences to test the robustness of their findings.
Summary updated