Abstract
Traditional entity linking (EL) systems rely on domain-specific metadata, such as alias tables and frequency statistics, which limits their ability to generalize to new, specialized domains. We introduce the zero-shot entity linking task, where models must link mentions to entities they have never seen before using only textual descriptions. We propose a two-stage pipeline using deep Transformer architectures and a novel Domain-Adaptive Pre-training (DAP) strategy. Our results on the new Zeshel benchmark demonstrate that robust language understanding and cross-attention are critical for generalizing across diverse domains.
The Challenge of Zero-Shot Entity Linking
Entity Linking (EL) is the process of connecting a textual mention (e.g., "The Burden spell") to its corresponding entry in a knowledge base (e.g., the Elder Scrolls wiki). While standard EL systems perform well in general domains like Wikipedia, they often fail in specialized contexts—such as legal documents or niche fan wikis—where structured metadata and alias tables are unavailable.
In the zero-shot setting, the model is trained on one set of "worlds" (domains) and tested on entirely different ones. This forces the system to move beyond memorization and instead rely on reading comprehension to resolve mentions based solely on entity descriptions.
Methodology: From Retrieval to Reading
The proposed system follows a two-stage approach:
- Candidate Generation: To handle large-scale entity dictionaries, an initial retrieval step using BM25 identifies the top 64 candidate entities based on text overlap between the mention context and entity descriptions.
- Candidate Ranking: The core of the model is a Full-Transformer ranker based on BERT. Unlike Siamese architectures that encode mentions and descriptions independently, the Full-Transformer uses cross-attention to jointly encode the mention-in-context and the entity description. This allows for fine-grained reasoning at every layer of the network.
To bridge the gap between training and testing domains, we introduce Domain-Adaptive Pre-training (DAP). This involves an additional Masked Language Modeling (MLM) step on the unlabeled text of the target domain before fine-tuning the model for the linking task.
The Zeshel Benchmark
To evaluate this task, the authors created Zeshel (Zero-Shot Entity Linking), a dataset derived from 16 distinct Wikias. The data is split such that the 8 worlds used for testing (e.g., Coronation Street, Lego, YuGiOh) share no entities or specific domain vocabulary with the 8 worlds used for training (e.g., Star Wars, Doctor Who). This setup ensures that models must learn a generalizable linking process rather than domain-specific shortcuts.
Results and the Power of Domain Adaptation
The Full-Transformer model significantly outperformed simpler architectures, achieving 76.06% accuracy compared to just 57.61% for a Siamese "Pool-Transformer." This confirms that cross-attention is vital for the "reading comprehension" required in zero-shot linking.
As shown in Figure 4, the DAP strategy consistently improves performance. A key finding is the strong correlation between a model's ability to predict missing words in a new domain (MLM accuracy) and its eventual success in linking entities within that domain. By chaining pre-training stages, the model reached its peak performance of 78.07% average accuracy.
Conclusion
Zero-shot entity linking highlights the importance of deep language understanding for robust information extraction. While the Full-Transformer and DAP provide a strong foundation, challenges remain—particularly in the initial retrieval stage and in handling "NIL" entities that do not exist in the dictionary. Future work will focus on improving zero-shot candidate generation and exploring document-level collective inference.
Summary updated