Introduction
Entity Linking (EL) is a fundamental task in natural language processing that involves mapping a textual mention (e.g., "the dark knight") to a specific entry in a knowledge base (e.g., the Batman film). In a zero-shot setting, the challenge is amplified: the model must link mentions to entities it has never encountered during training, relying entirely on short textual descriptions of those candidates.
The Challenge of Sequence Length
State-of-the-art EL models often utilize BERT-based architectures. However, standard BERT models are limited to a maximum sequence length of 512 tokens. In zero-shot EL, the model must process both the context surrounding the mention and the full description of the candidate entity. When these combined elements exceed 512 tokens, the model is forced to truncate data, often losing the critical information required for an accurate match.
Researchers found that in the Wikia zero-shot benchmark, approximately 50% of the data exceeds this 512-token limit. While "Long-BERT" variants exist to handle more tokens, they typically require computationally expensive pre-training from scratch to learn new position embeddings.
Proposed Method: Embedding-repeat
To address the length limitation without the cost of full re-training, the authors propose a two-stage pipeline consisting of candidate generation (via BM25) and candidate ranking (via BERT). The core innovation lies in the ranking stage with a method called Embedding-repeat ($E_{repeat}$).
Instead of randomly initializing position embeddings for tokens 513 through 1024, the researchers simply copy the pre-trained embeddings from positions 1 through 512. This strategy leverages BERT’s inherent "local bias"—its tendency to focus on immediate neighbors—allowing the model to maintain structural understanding across an extended sequence.
Key Results and Insights
The $E_{repeat}$ method achieved a new state-of-the-art (SOTA) on the Wikia zero-shot benchmark, raising accuracy from 76.06% to 79.08%. The impact was most pronounced on long-form data (sequences > 512 tokens), where accuracy jumped from 74.57% to 82.14%.
Key takeaways from the study include:
- Simplicity and Efficiency: Smart initialization of position embeddings is sufficient for downstream tasks, removing the need for costly long-form pre-training.
- Robustness: $E_{repeat}$ proved significantly more effective than random initialization ($E_{head}$) or using a constant embedding ($E_{constant}$) for extended positions.
- Information Balance: Accuracy drops similarly whether mention context or entity descriptions are truncated, suggesting both are equally vital for the linking task.
Conclusion and Future Work
By extending the Effective Reading Length of BERT through a simple embedding copy-and-paste strategy, the researchers demonstrated that long-range dependencies in zero-shot entity linking can be captured efficiently. Future research may explore whether this $E_{repeat}$ strategy generalizes to other long-form NLP tasks, such as document classification or long-form question answering, and how it scales to even longer sequences beyond 1024 tokens.
Summary updated