Probing for Bridging Inference in Transformer Language Models
Lists
Only you see this
Abstract
We probe pre-trained transformer language models for bridging inference. We first investigate individual attention heads in BERT and observe that attention heads at higher layers prominently focus on bridging relations in-comparison with the lower and middle layers, also, few specific attention heads concentrate consistently on bridging. More importantly, we consider language models as a whole in our second approach where bridging anaphora resolution is formulated as a masked token prediction task (Of-Cloze test). Our formulation produces optimistic results without any fine-tuning, which indicates that pre-trained language models substantially capture bridging inference. Our further investigation shows that the distance between anaphor-antecedent and the context provided to language models play an important role in the inference.
BibTeX
@inproceedings{pandit2021probing,
author = {Onkar Pandit and Yufang Hou},
title = {{Probing for Bridging Inference in Transformer Language Models}},
booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
year = {2021},
url = {https://openreview.net/forum?id=UHAGdX0ZWxDZ},
}Loading related content...