MSnet: A BERT-based Network for Gendered Pronoun Resolution

Zili WangACL 2019

TL;DR

This paper introduces MSnet, a novel neural network architecture that leverages pre-trained BERT to tackle gender bias in pronoun resolution. By employing an attention mechanism for contextual entity span representation and a triple-wise semantic similarity vector, MSnet achieves state-of-the-art results on the Gendered Ambiguous Pronouns (GAP) dataset, significantly reducing gender bias in this crucial NLP task.

What Problem Does This Paper Address?

Coreference resolution, the task of identifying mentions that refer to the same real-world entity, is fundamental to natural language understanding. However, many existing coreference resolution models exhibit gender bias, leading to inaccurate pronoun resolution when gender is ambiguous. This bias can perpetuate harmful stereotypes. To address this, the Gendered Ambiguous Pronouns (GAP) dataset was introduced, specifically designed to evaluate and mitigate gender bias in pronoun resolution. This paper aims to build a model that can effectively resolve gendered ambiguous pronouns without relying on gender stereotypes.

Methods

The core of this work is the MSnet (Mention Score Network), a neural network built upon the pre-trained BERT model. MSnet operates as a mention score classifier. Its key components include:

  • Span Representation Layer: This layer captures the contextual representation of entity spans. It utilizes two methods:
  • Meanpooling: A straightforward approach that averages the hidden states of tokens within a span.
  • Attention Mechanism: A novel, parameter-free attention mechanism that weights tokens within a span based on their contextual similarity to the pronoun. This is highlighted as being more space-efficient and explainable.
  • Similarity Layer: This layer computes a vector representing the triple-wise semantic similarity among the pronoun and candidate entities. It uses a feed-forward neural network that takes contextual representations of the pronoun and entities as input.
  • Mention Score Layer: This final layer computes a score for each candidate entity, indicating the likelihood that the pronoun refers to it. It combines the similarity vector with a distance encoding vector representing the positional difference between the pronoun and entities.

The training strategy involves a two-step fine-tuning approach: first, MSnet is trained with BERT frozen (feature-based approach), and then both BERT and MSnet are fine-tuned together with a small learning rate. This two-step process is designed to prevent the model from getting stuck in local minima.

Datasets

  • Gendered Ambiguous Pronouns (GAP) dataset: This is the primary dataset used for training and evaluation. It contains 4454 records and is officially split into development, test, and validation sets. For this work, the official test and validation sets were combined for training, and the development set was used as the test set.

Key Insights and Ideas

  • BERT's Power for Gendered Pronoun Resolution: The paper demonstrates that pre-trained language models like BERT are highly effective for tackling gender bias in pronoun resolution, outperforming previous approaches.
  • Parameter-Free Attention for Span Representation: The introduction of a parameter-free attention mechanism for contextual span representation is a novel contribution, offering efficiency and interpretability benefits.
  • Two-Step Fine-Tuning is Crucial: The proposed two-step fine-tuning strategy (feature-based training followed by joint fine-tuning) is essential for achieving optimal performance and avoiding local optima.
  • Premature Strategy Outperforms Mature: The "premature" fine-tuning strategy (training for underfitting in the first step) yielded better results than the "mature" strategy (training for proper fitting), suggesting that maintaining a more explorative optimization landscape is beneficial for the subsequent fine-tuning.
  • Larger BERT Models Improve Performance: Using the larger BERT-LARGE model consistently reduced the loss compared to BERT-BASE, indicating that increased model capacity is beneficial for this task.

Key Results

  • Stage 1 Performance: A variant of MSnet trained using the fine-tuning approach achieved a multi-class logarithmic loss of 0.3033 on the 5-fold cross-validation of the training set.
  • Stage 2 Performance: The same variant achieved a score of 0.17289 on the test dataset in stage 2 of the task, securing 2nd place.
  • Feature-based Approach: The best performing model in the feature-based approach achieved a loss of 0.3582±0.0435 on the 5-fold CV on train and 0.3349 on the test set.
  • Attention vs. Meanpooling: The attention mechanism for span representation reduced the loss by approximately 0.02 compared to meanpooling.

Limitations and Future Work

  • Hyper-parameter Tuning: The paper explores various hyper-parameters, including the number of BERT layers for representation and the dimension of the similarity vector, suggesting that these choices can impact performance.
  • Computational Cost: The fine-tuning approach, especially with larger BERT models, is computationally intensive, as evidenced by the long runtimes reported.
  • Future Work: The authors believe MSnet can serve as a strong baseline for gendered pronoun resolution and coreference resolution tasks. The availability of the code suggests potential for further research and development by the community.

Summary updated

BibTeX
@inproceedings{wang2019msnet,
  author = {Zili Wang},
  title = {{MSnet: A BERT-based Network for Gendered Pronoun Resolution}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure