Fill the GAP: Exploiting BERT for Pronoun Resolution

Kai-Chou Yang, Timothy Niven, Tzu Hsuan Chou, Hung-Yu KaoACL 2019

TL;DR

This paper presents a BERT-based ensemble model that achieved fourth place in the Gendered Ambiguous Pronouns (GAP) dataset competition. The authors found that BERT's representations are redundant and that specific layers are more optimal for pronoun resolution, offering insights into how to better leverage these powerful language models.

What Problem Does This Paper Address?

Pronoun resolution, the task of identifying which noun a pronoun refers to, is a fundamental challenge in natural language understanding. This is particularly difficult when pronouns are gender-ambiguous, as in the Gendered Ambiguous Pronouns (GAP) dataset. The GAP dataset is designed to test for gender bias by balancing male and female pronouns and features long, complex sentences from encyclopedic texts, often requiring world knowledge to resolve. For example, understanding that "She" refers to "Christine" in the provided example requires knowing that "never been mentioned again" implies death, a piece of world knowledge not explicitly stated in the sentence. This task is crucial for building more nuanced and accurate language understanding systems.

Methods

The authors developed an ensemble system that leverages BERT, a powerful pre-trained language model. Instead of using the last layer of BERT, they explored each layer to find the "optimal" layer for pronoun resolution, discovering that intermediate layers performed best. They also addressed BERT's redundancy by applying dropout directly to BERT features, which they liken to a regularization technique. Their core architecture is a multi-head natural language inference (NLI) encoder. This encoder projects contextual representations of the pronoun and candidate entities into lower-dimensional subspaces, models their interactions using techniques inspired by NLI, and then aggregates these interactions. The final prediction is an ensemble of models trained on different BERT variants (Base/Large, Cased/Uncased) and handcrafted features, combined using a densely connected neural network.

Datasets

The primary dataset used is the Gendered Ambiguous Pronouns (GAP) dataset. This dataset is balanced for gendered pronouns and is derived from encyclopedic text. The authors used it for both training and evaluation, splitting it into validation, development, and testing sets. They note that the dataset is relatively small, which influenced their reliance on transfer learning from BERT.

Key Insights and Ideas

  • BERT Redundancy: BERT's representations are highly redundant. The authors found that using only 70-80% of the features from the optimal BERT layer was sufficient for strong performance, suggesting that not all of BERT's learned information is equally critical for this task.
  • Optimal BERT Layer: The last hidden layer of BERT is not always the best for every task. For pronoun resolution, intermediate layers (specifically layer 8 for BERT Base and layer 19 for BERT Large) yielded better results.
  • Interaction Modeling is Key: Modeling the interactions between pronoun and entity representations, similar to natural language inference, significantly improved performance.
  • Dropout as Regularization: Applying dropout directly to BERT features acts as a form of regularization, helping to prevent overfitting and improve generalization, especially given the dataset's small size.
  • Attention Energies as Features: The difference in attention energy between the pronoun and candidate entities in optimal BERT layers shows a strong correlation with the correct label, suggesting these energies could be valuable features for future work.

Key Results

The proposed ensemble model achieved fourth place in the Kaggle GAP shared-task competition. The paper demonstrates significant performance gains by incrementally adding components: input dropout, interaction layers, projection, and handcrafted features. For instance, combining these elements with BERT Large resulted in a log loss of 0.24 on Stage 2 testing data, a substantial improvement over a baseline using only handcrafted features (0.35). The ensemble of BERT Base and BERT Large models achieved the best performance, with a log loss of 0.18 on Stage 2.

Limitations and Future Work

The authors acknowledge that world knowledge cases remain a significant limitation, as their model still struggles with examples requiring external knowledge not present in the text. Future work includes incorporating features derived from BERT's attention layers, which showed promise as interpretable signals for coreference resolution. They also aim to investigate using this attention signal for unsupervised and semi-supervised pronoun resolution, potentially reducing the need for labeled data.

Summary updated

BibTeX
@inproceedings{yang2019fill,
  author = {Kai-Chou Yang and Timothy Niven and Tzu Hsuan Chou and Hung-Yu Kao},
  title = {{Fill the GAP: Exploiting BERT for Pronoun Resolution}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure