Zero-shot Entity Linking with Efficient Long Range Sequence Modeling

Zonghai Yao, Liangliang Cao, Huapu Pan

Introduction

Entity Linking (EL) is a fundamental task in natural language processing that involves mapping a textual mention (e.g., "the dark knight") to a specific entry in a knowledge base (e.g., the Batman film). In a zero-shot setting, the challenge is amplified: the model must link mentions to entities it has never encountered during training, relying entirely on short textual descriptions of those candidates.

The Challenge of Sequence Length

State-of-the-art EL models often utilize BERT-based architectures. However, standard BERT models are limited to a maximum sequence length of 512 tokens. In zero-shot EL, the model must process both the context surrounding the mention and the full description of the candidate entity. When these combined elements exceed 512 tokens, the model is forced to truncate data, often losing the critical information required for an accurate match.

Figure 1: Comparison of prediction results between small and large Effective Reading Lengths (ERLength). Truncating the context leads to an incorrect match, while including more context allows the model to correctly identify the entity.

Researchers found that in the Wikia zero-shot benchmark, approximately 50% of the data exceeds this 512-token limit. While "Long-BERT" variants exist to handle more tokens, they typically require computationally expensive pre-training from scratch to learn new position embeddings.

Proposed Method: Embedding-repeat

To address the length limitation without the cost of full re-training, the authors propose a two-stage pipeline consisting of candidate generation (via BM25) and candidate ranking (via BERT). The core innovation lies in the ranking stage with a method called Embedding-repeat ($E_{repeat}$).

Instead of randomly initializing position embeddings for tokens 513 through 1024, the researchers simply copy the pre-trained embeddings from positions 1 through 512. This strategy leverages BERT’s inherent "local bias"—its tendency to focus on immediate neighbors—allowing the model to maintain structural understanding across an extended sequence.

Figure 5: The Embedding-repeat strategy ($E_{repeat}$) compared to other initialization methods. By copying existing embeddings, the model can be fine-tuned for longer sequences without expensive intermediate pre-training.

Key Results and Insights

The $E_{repeat}$ method achieved a new state-of-the-art (SOTA) on the Wikia zero-shot benchmark, raising accuracy from 76.06% to 79.08%. The impact was most pronounced on long-form data (sequences > 512 tokens), where accuracy jumped from 74.57% to 82.14%.

Key takeaways from the study include:

  • Simplicity and Efficiency: Smart initialization of position embeddings is sufficient for downstream tasks, removing the need for costly long-form pre-training.
  • Robustness: $E_{repeat}$ proved significantly more effective than random initialization ($E_{head}$) or using a constant embedding ($E_{constant}$) for extended positions.
  • Information Balance: Accuracy drops similarly whether mention context or entity descriptions are truncated, suggesting both are equally vital for the linking task.

Conclusion and Future Work

By extending the Effective Reading Length of BERT through a simple embedding copy-and-paste strategy, the researchers demonstrated that long-range dependencies in zero-shot entity linking can be captured efficiently. Future research may explore whether this $E_{repeat}$ strategy generalizes to other long-form NLP tasks, such as document classification or long-form question answering, and how it scales to even longer sequences beyond 1024 tokens.

Summary updated

Abstract

This paper considers the problem of zero-shot entity linking, in which a link in the test time may not present in training. Following the prevailing BERT-based research efforts, we find a simple yet effective way is to expand the long-range sequence modeling. Unlike many previous methods, our method does not require expensive pre-training of BERT with long position embedding. Instead, we propose an efficient position embeddings initialization method called Embedding-repeat, which initializes larger position embeddings based on BERT-Base. On Wikia's zero-shot EL dataset, our method improves the SOTA from 76.06% to 79.08%, and for its long data, the corresponding improvement is from 74.57% to 82.14%. Our experiments suggest the effectiveness of long-range sequence modeling without retraining the BERT model.
BibTeX
@inproceedings{yao2020zero,
  author = {Zonghai Yao and Liangliang Cao and Huapu Pan},
  title = {{Zero-shot Entity Linking with Efficient Long Range Sequence Modeling}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  doi = {10.18653/v1/2020.findings-emnlp.228},
}
Figure