Probing for Bridging Inference in Transformer Language Models

Onkar Pandit, Yufang Hou

Abstract

We probe pre-trained transformer language models for bridging inference. We first investigate individual attention heads in BERT and observe that attention heads at higher layers prominently focus on bridging relations in-comparison with the lower and middle layers, also, few specific attention heads concentrate consistently on bridging. More importantly, we consider language models as a whole in our second approach where bridging anaphora resolution is formulated as a masked token prediction task (Of-Cloze test). Our formulation produces optimistic results without any fine-tuning, which indicates that pre-trained language models substantially capture bridging inference. Our further investigation shows that the distance between anaphor-antecedent and the context provided to language models play an important role in the inference.
BibTeX
@inproceedings{pandit2021probing,
  author = {Onkar Pandit and Yufang Hou},
  title = {{Probing for Bridging Inference in Transformer Language Models}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=UHAGdX0ZWxDZ},
}

Loading related content...

Figure