TL;DR
This paper tackles the challenge of gender bias in pronoun resolution systems by reformulating the task as an extractive question answering problem using BERT. Their novel approach significantly improves performance and reduces gender bias, even without explicit knowledge of candidate antecedents, and achieves competitive results on a challenging gender-balanced dataset.
What Problem Does This Paper Address?
Resolving ambiguous pronouns (like "he," "she," "it") to their correct referents is a fundamental task in Natural Language Understanding (NLU). However, existing state-of-the-art coreference resolution systems have been shown to exhibit a significant gender bias. This means they tend to perform worse or make more errors when dealing with pronouns referring to certain genders, or when the antecedents themselves have gendered names. To address this, a new gender-balanced dataset was released, aiming to encourage the development of fairer models. The challenge lies in achieving high accuracy on this dataset, especially given the limited annotated labels, and ensuring the models are not implicitly learning or perpetuating gender stereotypes.
Methods
The core innovation of this paper is framing the pronoun resolution task as an extractive question answering (QA) problem. Instead of traditional coreference resolution approaches, they leverage the power of BERT, a pre-trained language model.
Here's how it works:
- QA Formulation (CorefQA): The input is a Wikipedia passage. The "question" is a short context window (up to 5 words) around the ambiguous pronoun. The "answer" is the antecedent of the pronoun, which is expected to be found within the passage. This formulation is inspired by the SQuAD dataset. A key advantage is that this core QA model doesn't require prior knowledge of candidate antecedents (A and B). It learns to extract the correct antecedent directly from the text.
- Probability Estimation (CorefQAExt): To output probabilities for A, B, and "neither" (N), they extend the CorefQA model. They extract span logits corresponding to candidate antecedents A and B, along with overall sequence logits. These are fed into a multi-class logistic regression layer to produce the final probabilities.
- Multiple Choice (CorefMulti): This approach treats the task as a multiple-choice problem, similar to the SWAG dataset. For each example, they construct input sequences by concatenating the passage with potential antecedents (A, B, or "neither") and feed them to BERT for classification.
- Sequence Classification (CorefSeq): This is a more standard approach where the model directly classifies the pronoun's antecedent as A, B, or N based on the passage's token embeddings.
- Ensemble: The authors combine the predictions of CorefQAExt, CorefMulti, and CorefSeq models to achieve even better performance.
Datasets
The primary dataset used is the GAP dataset (Gendered Ambiguous Pronoun dataset) released by Webster et al. (2018). This dataset was specifically created to be gender-balanced, containing an equal number of male and female examples. The Kaggle competition for this dataset was conducted in two stages, with different splits for training, validation, and testing. The authors also mention using a pre-trained BERT model, which was trained on a massive corpus of text, including Wikipedia.
Key Insights and Ideas
- QA as a powerful abstraction: Reformulating pronoun resolution as a QA task proves highly effective, demonstrating that BERT can learn to identify antecedents even without explicit candidate information.
- Unbiased predictions from pre-trained models: The study shows that fine-tuning pre-trained BERT on a gender-balanced dataset can lead to unbiased predictions, suggesting that the bias might stem more from the training data than the model architecture itself.
- Context window is key: Using the pronoun's immediate context window as the "question" in the QA formulation helps the model disambiguate when multiple instances of a pronoun might appear in a passage.
- Ensembling for robustness: Combining different modeling approaches (QA, multiple choice, sequence classification) leads to significant performance gains, highlighting the benefit of diverse perspectives on the task.
Key Results
The proposed QA formulation (CorefQAExt) achieved a significant improvement of 22.2% absolute F1 score over the existing baseline on the gender-balanced dataset. The ensemble model, combining CorefQAExt, CorefMulti, and CorefSeq, further boosted performance, reaching an overall F1 score of 90.2 and an absolute improvement of 23.3% over the baseline. This ensemble model ranked 9th on the final official leaderboard of the ACL workshop shared task. Notably, the CorefQA model, which doesn't use candidate antecedent information, still outperformed the baseline by 21.4% in F1 score. The CorefMulti model showed particular robustness to bias, achieving a bias score of 0.99.
Limitations and Future Work
- Computational Cost: Fine-tuning BERT models, especially for large datasets and multiple epochs, is computationally expensive.
- Unsupervised Fine-tuning: Initial experiments with unsupervised fine-tuning of BERT on Wikipedia texts from the dataset did not yield significant gains, possibly because BERT is already pre-trained on similar data.
- Entity Embeddings: Attempts to incorporate additional entity embeddings into the CorefMulti model did not improve performance within the competition deadline, but this is suggested as a promising future direction.
- Appending Title: Appending the Wikipedia page title to the input sequence for CorefQAExt slightly worsened performance.
- Future Directions: The authors suggest applying their techniques to other coreference resolution datasets (like Winograd Schema Challenge, Winogender), exploring alternative contextual representations, and extending the work to other languages. They also propose a more in-depth analysis of the gains from BERT representations.
Summary updated