Zero-Shot Entity Linking by Reading Entity Descriptions

Lajanugen Logeswaran, Ming-Wei Chang, Kenton Lee, Kristina Toutanova, Jacob Devlin, Honglak Lee

Abstract

We present the zero-shot entity linking task, where mentions must be linked to unseen entities without in-domain labeled data. The goal is to enable robust transfer to highly specialized domains, and so no metadata or alias tables are assumed. In this setting, entities are only identified by text descriptions, and models must rely strictly on language understanding to resolve the new entities. First, we show that strong reading comprehension models pre-trained on large unlabeled data can be used to generalize to unseen entities. Second, we propose a simple and effective adaptive pre-training strategy, which we term domain-adaptive pre-training (DAP), to address the domain shift problem associated with linking unseen entities in a new domain. We present experiments on a new dataset that we construct for this task and show that DAP improves over strong pre-training baselines, including BERT.

Introduction

Entity Linking (EL) is the fundamental task of mapping a mention in text to its corresponding entry in a knowledge base. While traditional EL systems have achieved high performance, they often rely on "crutches" such as alias tables, structured metadata, and global frequency statistics. These dependencies make them brittle when applied to specialized or emerging domains—such as internal corporate documentation or niche fan wikis—where such metadata is unavailable.

We propose a "zero-shot" setting for EL where a model must link mentions to entities it has never seen during training, relying solely on text descriptions. This shifts the problem from a pattern-matching task to a deep language understanding task.

The Zero-Shot Challenge

In the zero-shot setting, the model is trained on a set of source domains and evaluated on completely disjoint target domains. For example, a model might learn to link entities in the Elder Scrolls universe and then be expected to link entities in a Lego or Star Trek context without any additional labeled training data for those new worlds.

Overview of the zero-shot entity linking task across disjoint domains.

Because the entities in the test set are unseen, the model cannot rely on memorized entity names or statistical priors. It must instead "read" the context of the mention and the entity descriptions to find the correct match.

Methodology

Our approach utilizes a two-stage pipeline designed for both efficiency and accuracy:

  1. Candidate Generation: To handle large-scale entity dictionaries, we first use BM25, a classic information retrieval algorithm, to retrieve the top 64 candidate entities based on the lexical similarity between the mention context and entity descriptions.
  2. Candidate Ranking (The Full-Transformer): We employ a Cross-Encoder architecture based on a 12-layer Transformer (BERT). Unlike Bi-Encoders that encode mentions and entities separately, our model concatenates the mention context and the entity description into a single sequence. This allows for fine-grained cross-attention at every layer, which is crucial for resolving subtle ambiguities.

Domain-Adaptive Pre-training (DAP)

To mitigate the domain shift between training and testing, we introduce Domain-Adaptive Pre-training (DAP). Before fine-tuning on the EL task, the model undergoes an additional phase of Masked Language Modeling (MLM) using the unlabeled text descriptions from the target domain. This allows the model to adapt its internal representations to the specific vocabulary and semantics of the new domain.

The Zeshel Benchmark

To evaluate this task, we introduce the Zeshel (Zero-Shot Entity Linking) dataset. It is constructed from 16 distinct sub-wikis from Wikia (Fandom), covering diverse topics like American Football, Doctor Who, and YuGiOh. The dataset is split such that the training, validation, and test sets contain entirely different "worlds," ensuring that the model is truly tested on its ability to generalize to unseen entities.

Results and Analysis

Our experiments demonstrate that the Cross-Encoder architecture significantly outperforms simpler pooling-based models, proving that deep cross-attention is essential for zero-shot linking. Furthermore, the inclusion of DAP provides a consistent performance boost.

Impact of Domain-Adaptive Pre-training (DAP) on linking accuracy.

As shown in the results, DAP improved normalized accuracy by approximately 2% over a strong BERT baseline, reaching a top performance of 77.05% on unseen test domains. We also observed a strong correlation between the model's accuracy on the target domain's MLM task and its subsequent performance on the entity linking task, validating the effectiveness of the adaptive pre-training strategy.

Conclusion and Future Work

This work establishes that reading comprehension models can effectively link entities in a zero-shot setting by relying on text descriptions. While DAP provides a significant boost, several challenges remain. The current BM25 candidate generation stage is a bottleneck, with only 68% recall on test data. Future research will focus on improving this retrieval stage, handling "NIL" entities (mentions with no corresponding entry), and incorporating broader document-level context to improve consistency across multiple mentions.

Summary updated

Abstract

We present the zero-shot entity linking task, where mentions must be linked to unseen entities without in-domain labeled data. The goal is to enable robust transfer to highly specialized domains, and so no metadata or alias tables are assumed. In this setting, entities are only identified by text descriptions, and models must rely strictly on language understanding to resolve the new entities. First, we show that strong reading comprehension models pre-trained on large unlabeled data can be used to generalize to unseen entities. Second, we propose a simple and effective adaptive pre-training strategy, which we term domain-adaptive pre-training (DAP), to address the domain shift problem associated with linking unseen entities in a new domain. We present experiments on a new dataset that we construct for this task and show that DAP improves over strong pre-training baselines, including BERT. The data and code are available at https://github.com/lajanugen/zeshel.
BibTeX
@inproceedings{logeswaran2019zero,
  author = {Lajanugen Logeswaran and Ming-Wei Chang and Kenton Lee and Kristina Toutanova and Jacob Devlin and Honglak Lee},
  title = {{Zero-Shot Entity Linking by Reading Entity Descriptions}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=t3skUabvohQ},
}
Figure