Zero-Shot Entity Linking by Reading Entity Descriptions

Lajanugen Logeswaran, Ming-Wei Chang, Kenton Lee, Kristina Toutanova, Jacob Devlin, Honglak Lee

Abstract

Traditional entity linking (EL) systems rely on domain-specific metadata, such as alias tables and frequency statistics, which limits their ability to generalize to new, specialized domains. We introduce the zero-shot entity linking task, where models must link mentions to entities they have never seen before using only textual descriptions. We propose a two-stage pipeline using deep Transformer architectures and a novel Domain-Adaptive Pre-training (DAP) strategy. Our results on the new Zeshel benchmark demonstrate that robust language understanding and cross-attention are critical for generalizing across diverse domains.

The Challenge of Zero-Shot Entity Linking

Entity Linking (EL) is the process of connecting a textual mention (e.g., "The Burden spell") to its corresponding entry in a knowledge base (e.g., the Elder Scrolls wiki). While standard EL systems perform well in general domains like Wikipedia, they often fail in specialized contexts—such as legal documents or niche fan wikis—where structured metadata and alias tables are unavailable.

In the zero-shot setting, the model is trained on one set of "worlds" (domains) and tested on entirely different ones. This forces the system to move beyond memorization and instead rely on reading comprehension to resolve mentions based solely on entity descriptions.

The zero-shot entity linking setup involves training on a set of source domains and testing on completely unseen target domains.

Methodology: From Retrieval to Reading

The proposed system follows a two-stage approach:

  1. Candidate Generation: To handle large-scale entity dictionaries, an initial retrieval step using BM25 identifies the top 64 candidate entities based on text overlap between the mention context and entity descriptions.
  2. Candidate Ranking: The core of the model is a Full-Transformer ranker based on BERT. Unlike Siamese architectures that encode mentions and descriptions independently, the Full-Transformer uses cross-attention to jointly encode the mention-in-context and the entity description. This allows for fine-grained reasoning at every layer of the network.

To bridge the gap between training and testing domains, we introduce Domain-Adaptive Pre-training (DAP). This involves an additional Masked Language Modeling (MLM) step on the unlabeled text of the target domain before fine-tuning the model for the linking task.

The Zeshel Benchmark

To evaluate this task, the authors created Zeshel (Zero-Shot Entity Linking), a dataset derived from 16 distinct Wikias. The data is split such that the 8 worlds used for testing (e.g., Coronation Street, Lego, YuGiOh) share no entities or specific domain vocabulary with the 8 worlds used for training (e.g., Star Wars, Doctor Who). This setup ensures that models must learn a generalizable linking process rather than domain-specific shortcuts.

Results and the Power of Domain Adaptation

The Full-Transformer model significantly outperformed simpler architectures, achieving 76.06% accuracy compared to just 57.61% for a Siamese "Pool-Transformer." This confirms that cross-attention is vital for the "reading comprehension" required in zero-shot linking.

Performance of DAP across different pre-training configurations and the correlation between target domain MLM accuracy and entity-linking performance.

As shown in Figure 4, the DAP strategy consistently improves performance. A key finding is the strong correlation between a model's ability to predict missing words in a new domain (MLM accuracy) and its eventual success in linking entities within that domain. By chaining pre-training stages, the model reached its peak performance of 78.07% average accuracy.

Conclusion

Zero-shot entity linking highlights the importance of deep language understanding for robust information extraction. While the Full-Transformer and DAP provide a strong foundation, challenges remain—particularly in the initial retrieval stage and in handling "NIL" entities that do not exist in the dictionary. Future work will focus on improving zero-shot candidate generation and exploring document-level collective inference.

Summary updated

Abstract

We present the zero-shot entity linking task, where mentions must be linked to unseen entities without in-domain labeled data. The goal is to enable robust transfer to highly specialized domains, and so no metadata or alias tables are assumed. In this setting, entities are only identified by text descriptions, and models must rely strictly on language understanding to resolve the new entities. First, we show that strong reading comprehension models pre-trained on large unlabeled data can be used to generalize to unseen entities. Second, we propose a simple and effective adaptive pre-training strategy, which we term domain-adaptive pre-training (DAP), to address the domain shift problem associated with linking unseen entities in a new domain. We present experiments on a new dataset that we construct for this task and show that DAP improves over strong pre-training baselines, including BERT. The data and code are available at https://github.com/lajanugen/zeshel.
BibTeX
@inproceedings{logeswaran2019zero,
  author = {Lajanugen Logeswaran and Ming-Wei Chang and Kenton Lee and Kristina Toutanova and Jacob Devlin and Honglak Lee},
  title = {{Zero-Shot Entity Linking by Reading Entity Descriptions}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=t3skUabvohQ},
}
Figure