BERT Masked Language Modeling for Co-reference Resolution

Felipe Alfaro, Marta R. Costa-jussà, José A. R. FonollosaACL 2019

TL;DR

This paper explores using BERT's masked language modeling capabilities for co-reference resolution, specifically the gendered pronoun resolution task. They introduce a "Name Replacement" technique to mitigate biases from name frequency in BERT's training data, achieving competitive results on the GAP dataset.

What Problem Does This Paper Address?

Co-reference resolution aims to identify which noun or phrase a pronoun refers to. A significant challenge in this field, particularly for gendered pronouns, is dataset bias. Datasets like OntoNotes, commonly used for training, disproportionately feature male entities, leading to models that inherit this bias. The Gendered Pronoun Resolution task, specifically the one discussed here, aims to address this by using a gender-neutral dataset (GAP) and encouraging gender-neutral models. The authors tackle the problem of how to effectively train a model on such a dataset, especially when common language models might have inherent biases due to name frequency in their pre-training corpora.

Methods

The core of the proposed solution leverages BERT, a powerful pre-trained language model. Specifically, the authors adapt BERT's masked language modeling (MLM) objective for co-reference resolution.

  • Masked Language Modeling Adaptation: Instead of predicting a masked word in a general context, the pronoun in question is masked. BERT then predicts the probability of different names (or "none") filling that masked position.
  • Name Replacement Technique: To combat potential biases from name frequency in BERT's training data, the authors implement a "Name Replacement" strategy. Common English names are substituted for the original names in the dataset. This has two key benefits:
  • Improved BERT Performance: Common names are more likely to be well-represented in BERT's pre-training corpus, leading to better predictions.
  • Tokenization Benefits: By choosing names that are not split into multiple word pieces by BERT's tokenizer, they ensure that the pronoun and the candidate names are treated as single tokens, simplifying the comparison process.
  • Two Models:
    • Model 1: A straightforward classification approach where the probabilities of the two candidate names and "none" are directly used.
  • Model 2: This model enhances Model 1 by creating two versions of the input text where the candidate names are swapped. The probabilities from both versions are then averaged before applying the final classification. This aims to further reduce the impact of any lingering bias from the chosen replacement names.

Datasets

  • GAP Dataset: This dataset, created by Google AI Language, was used for training and evaluation. It contains 8908 co-reference labeled pairs sampled from Wikipedia, balanced between male and female representations. Each entry includes a text, a pronoun, its offset, and two names (name A and name B) that the pronoun might refer to, or "none." The dataset notably lacks neutral pronouns like "it" or "they."
  • Competition Stages: For the competition, two stages were involved:
  • Stage 1: Used the development set from the GAP repository for training, combining GAP validation and testing sets.
  • Stage 2: Used data available through Kaggle for submission, with correct labels not yet released. Training for Stage 2 combined GAP development, testing, and validation sets.

Key Insights and Ideas

  • BERT's Adaptability: The paper demonstrates that BERT's masked language modeling, typically used for general word prediction, can be effectively repurposed for a specific classification task like co-reference resolution.
  • Name Frequency Bias is Real: The "Name Replacement" technique highlights that even with a balanced dataset, the underlying pre-trained model can carry biases from its training data, particularly concerning name frequency.
  • Common Names Help: Using common English names as replacements significantly improved performance, likely due to their better representation in BERT's pre-training corpus.
  • Tokenization Matters: Ensuring that candidate names are single tokens simplifies the model's task of comparing them to the masked pronoun.
  • Averaging Swapped Names (Model 2) is Beneficial: Model 2, which averages predictions from swapped name inputs, showed improved results, suggesting it effectively mitigates some of the remaining name-specific biases.
  • The "None" Class is Challenging: Both models struggled with the "none" class, likely because it's less frequent in the dataset and the model is inherently geared towards identifying name references rather than their absence.

Key Results

  • Competitive Performance: Both models achieved strong results on the GAP dataset. For Stage 1 testing, Model 1 achieved an average F1 score of 0.84, and Model 2 achieved 0.80.
  • Name Replacement Impact: Table 7 shows a notable improvement in accuracy and a significant reduction in loss for Model 1 when using name replacement (0.838 accuracy, 0.4423 loss) compared to using original names (0.782 accuracy, 0.7021 loss).
  • Stage 2 Improvement: Results for Stage 2 were generally better than Stage 1, with Model 2 outperforming Model 1 in this stage (loss of 0.30151 for Model 2 vs. 0.31441 for Model 1).
  • Competition Ranking: The authors placed 46th in the official Kaggle competition, with their second model achieving a loss around 0.301.

Limitations and Future Work

  • Gendered Nature: The "Name Replacement" technique relies on knowing the gender of the names, limiting its applicability to entities with defined genders.
  • "None" Class Difficulty: The models struggled with correctly identifying cases where the pronoun refers to neither of the provided names. The authors suggest further investigation into the characteristics of these examples and how to better train the model to identify them, especially given their lower frequency.
  • Fine-tuning BERT Weights: The paper mentions that fine-tuning the actual BERT model weights could be a direction for future improvement.

Summary updated

BibTeX
@inproceedings{alfaro2019bert,
  author = {Felipe Alfaro and Marta R. Costa-jussà and José A. R. Fonollosa},
  title = {{BERT Masked Language Modeling for Co-reference Resolution}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure