TL;DR
This paper tackles the challenge of gender bias in pronoun resolution tasks by leveraging pre-trained BERT models. They demonstrate that fine-tuning BERT significantly improves performance and, crucially, helps to balance predictions between masculine and feminine pronouns, outperforming existing methods.
What Problem Does This Paper Address?
Pronoun resolution, a core task in natural language processing (NLP), aims to identify which noun a pronoun refers to in a sentence. While traditional coreference resolution systems have achieved good performance on general datasets, they often exhibit a significant gender bias. This means they perform worse for one gender than the other, a problem highlighted by the Gendered Ambiguous Pronouns (GAP) dataset. This dataset, specifically designed with balanced gendered pronoun-name pairs, revealed that many state-of-the-art systems struggled and showed a performance gap between genders. The authors investigate whether transfer learning, specifically using pre-trained language models like BERT, can not only improve overall performance but also mitigate this gender bias.
Methods
The authors explore two main approaches for leveraging BERT:
- Fine-tuned BERT: This involves taking a pre-trained BERT model and adding custom "top layers" that are trained along with the BERT model itself. They experimented with two types of top layers:
- MLP-top: This layer concatenates the BERT embeddings for all mentions (pronoun, potential antecedents A and B) and feeds them into a multi-layer perceptron (MLP).
- Positional-top: This layer maps BERT embeddings to scalars, extracts the relevant scalar for each mention, and then uses a softmax layer for classification.
- BERT as a Feature Extractor: In this approach, BERT's contextual embeddings are used as features, but the BERT model's weights are not updated during training. Instead, these embeddings are fed into separate classifiers like Support Vector Machines (SVM) or Bidirectional Attention Flow (BIDAF) networks.
To address potential overfitting with BERT's high-dimensional embeddings (1024 dimensions), they also employed a dimension reduction technique using a linear autoencoder, inspired by PPA-PCA-PPA, to reduce vectors to 256 dimensions. For their final submission, they used an ensemble learning approach, specifically stacking, where multiple base models (including the ones described above) make predictions, and a meta-classifier (logistic regression) makes the final decision.
Datasets
The primary dataset used is the GAP (Gendered Ambiguous Pronouns) dataset. This dataset consists of:
- 2000 development pairs (used for training in this paper, contrary to the official evaluation split).
- 2000 test pairs (used for official evaluation).
- 454 validation pairs (used for parameter tuning).
They also mention using the Caliskan dataset for a scientific statistical experiment to study gender bias in word and sentence level embeddings.
Key Insights and Ideas
- BERT's Power for Pronoun Resolution: Fine-tuning BERT models significantly outperforms BERT used purely as a feature extractor, indicating that adapting BERT's internal representations is crucial for this task.
- Dimension Reduction is Key: Reducing the dimensionality of BERT embeddings (from 1024 to 256) greatly enhances SVM performance and helps mitigate overfitting, bridging the gap between training and testing performance.
- Data Augmentation Nuances: Augmenting the BIDAF model with additional "neither" examples surprisingly worsened its performance, suggesting a distribution mismatch between training and testing data. However, this augmented data still proved beneficial for the final ensemble model.
- Ensemble Learning Boost: Stacking multiple models, including fine-tuned BERT variants, provided a substantial improvement in overall performance, highlighting the benefit of combining diverse predictive strengths.
- No Evidence of Gender Bias in Contextual Embeddings: Unlike older word embeddings (GloVe), contextual embeddings from ELMO, BERT, and fine-tuned BERT showed no significant gender bias in WEAT/SEAT tests. This is attributed to the context-dependent nature of word representations in these models.
Key Results
- The authors' final ensemble model achieved a multi-class logarithmic loss of 0.208, ranking 14th out of 838 teams in the shared task.
- Fine-tuned BERT models (BERT-pos and BERT-mlp) significantly outperformed BERT-as-feature-extractor models. For instance, BERT-pos achieved an F1 score of 86.5, while BERT-mlp achieved 86.1, both outperforming previous state-of-the-art results by a considerable margin (e.g., 23.3% F1 improvement over Lee et al., 2017).
- The ensemble model achieved the best overall F1 score of 88.0.
- WEAT and SEAT experiments showed that while GloVe embeddings exhibited significant gender bias, contextual embeddings from BERT and fine-tuned BERT did not show evidence of such bias.
Limitations and Future Work
- Data Augmentation Mismatch: The authors noted that data augmentation for BIDAF led to a distribution mismatch, impacting its standalone performance.
- Ensemble Model Complexity: While effective, the ensemble approach involves training and managing multiple base models.
- Future Work: The authors suggest investigating various transfer learning structures on top of pre-trained BERT to enhance the stability of the fine-tuning process, as they observed that performance can be sensitive to the initial random state. They also aim to build more robust models.
Summary updated