Abstract
Recently, a zero-shot entity linking task is introduced to challenge the generalization ability of entity linking models. In this task, mentions must be linked to unseen entities and only the textual information is available. In order to make full use of the documents, previous work has proposed a BERT-based model which can only take fixed length of text as input. However, the key information for entity linking may exist in nearly everywhere of the documents thus the proposed model cannot capture them all. To leverage more textual information and enhance text understanding capability, we propose a bidirectional multi-paragraph reading model for the zero-shot entity linking task. Firstly, the model treats the mention context as a query and matches it with multiple paragraphs of the entity description documents. Then, the mention-aware entity representation obtained from the first step is used as a query to match multiple paragraphs in the document containing the mention through an entity-mention attention mechanism. In particular, a new pre-training strategy is employed to strengthen the representative ability. Experimental results show that our bidirectional model can capture long-range context dependencies and outperform the baseline model by 3-4% in terms of accuracy.
Introduction
Entity Linking (EL) is the fundamental task of mapping a mention in text to its corresponding entry in a knowledge base. While traditional EL methods leverage frequency statistics and metadata from well-structured sources like Wikipedia, zero-shot entity linking presents a significantly harder challenge. In this setting, models must link mentions to entities in specialized domains (e.g., fan wikis for Star Wars or Lego) that were never encountered during training, relying solely on textual descriptions.
A major bottleneck for current BERT-based models is the fixed input length limit. Crucial evidence for linking a mention is often scattered across multiple paragraphs or located far from the mention itself. As illustrated in Figure 1, identifying "The Mummy" requires connecting information from different parts of the character's description and the mention's context.
Methodology
To address the limitations of local context, we propose the Bidirectional Multi-paragraph Reading (Bi-MPR) model. Unlike standard models that truncate long documents, Bi-MPR employs a two-stage "reading" process:
- Forward Matching (Uni-MPR): The model uses the mention context as a query to scan multiple paragraphs of the candidate entity's description. An inter-paragraph attention mechanism is used to weigh and aggregate the most relevant information from the entity document.
- Backward Matching: The resulting "mention-aware entity representation" then serves as a query to scan other paragraphs in the document where the mention appeared. This allows the model to find supporting clues that are not present in the immediate sentence surrounding the mention.
Furthermore, we introduce Whole Entity Masking (WEM) during pre-training. By masking entire entity names rather than random tokens, the model is forced to learn deeper contextual representations to predict the missing entity.
Experiments and Results
The model was evaluated on a zero-shot dataset derived from Wikia, using 8 domains for training and 8 entirely different domains for testing. This setup ensures that the model's ability to generalize to unseen entities is rigorously tested.
Experimental results demonstrate that Bi-MPR achieves a Micro Accuracy of 80.1%, a significant improvement over the 76.5% achieved by the standard BERT baseline. The performance gains were most pronounced in domains with long, complex documents, confirming the effectiveness of the multi-paragraph approach.
Conclusion
The Bi-MPR model effectively captures long-range context dependencies by treating entity linking as a bidirectional reading comprehension task. By leveraging information from multiple paragraphs in both the mention and entity documents, the model overcomes the truncation issues inherent in standard transformer architectures. Future work will focus on optimizing inference speed and improving performance on mentions with low textual overlap with their target entities.
Summary updated