Unified Examination of Entity Linking in Absence of Candidate Sets

Nicolas Ong, Hassan Shavarani, Anoop SarkarNAACL 2024 · DOI

TL;DR

This paper introduces a unified framework to evaluate modern entity linking systems, revealing their significant, often overlooked, dependence on pre-defined candidate sets. By systematically removing these sets, the authors demonstrate that many state-of-the-art systems become brittle, highlighting a critical area for improvement in building more robust and generalizable entity linking solutions.

What Problem Does This Paper Address?

Entity linking (EL) is the task of identifying mentions of entities in text and linking them to their corresponding entries in a knowledge base. While EL systems have seen remarkable progress, a significant challenge lies in their reliance on pre-generated "candidate sets" – a curated list of potential entities for each mention. These candidate sets are often assumed to be readily available and of high quality, especially when linking to well-established knowledge bases like Wikipedia. However, this assumption breaks down in many real-world scenarios where such pre-built sets are scarce or non-existent. The paper addresses this gap by providing a unified evaluation framework to systematically assess how entity linking systems perform without these crucial candidate sets, revealing their underlying robustness and limitations.

Methods

The core of this paper is a novel unified black-box evaluation framework built upon GERBIL and gerbil_connect. This framework allows for a consistent comparison of various entity linking systems without needing to delve into their internal architectures. The authors then conduct an "ablation study," a key experimental technique where a specific component (in this case, the candidate sets) is systematically removed or altered to observe its impact on performance.

To evaluate systems in the absence of candidate sets, they propose two main strategies:

  1. Candidate-Set-Independent Setting: For models that inherently support this, they disable the candidate set mechanism.
  2. In-Domain Vocabulary as Substitute: For models that cannot operate without a candidate set, they replace the pre-built sets with the entire in-domain vocabulary of the benchmark dataset. This ensures that the models still have a set of potential entities to choose from, albeit a much larger and less curated one.

The paper also details several necessary modifications to the evaluated systems to fit into this unified framework, including input tokenization, document splitting for models with length limits, and token-to-character annotation conversion.

Datasets

The primary dataset used for evaluation is CoNLL/AIDA, a standard benchmark for entity linking. The authors also utilize a newly annotated dataset, AIDA/testc, which contains fresh annotations on news stories from 2020 and includes 924 novel entities, serving as a more challenging test for generalization.

Key Insights and Ideas

  • Over-reliance on Candidate Sets: The most striking finding is the extreme dependence of most current entity linking systems on pre-built candidate sets. Without them, performance plummets, demonstrating that many systems are not robust enough for real-world applications where such sets are not readily available.
  • Generative Models Show More Resilience: Models that can generate candidate sets or operate without them (e.g., generative models) exhibit greater robustness compared to traditional mention-entity similarity-based approaches.
  • Impact on Different Error Types: The absence of candidate sets significantly increases the rate of incorrect entity predictions, highlighting that candidate sets play a crucial role in guiding the disambiguation process.
  • Trade-off with Larger Sets: While using the entire in-domain vocabulary as a candidate set can substitute for pre-built sets, it leads to significantly longer inference times for many models.
  • Need for Robustness: The study underscores the need for developing entity linking systems that are less reliant on curated candidate sets to ensure their applicability and reliability in diverse, real-world scenarios.

Key Results

  • Several models experienced a dramatic drop in performance, with some resulting in Micro-F1 scores close to 0 when candidate sets were removed.
  • Models that rely heavily on mention-entity similarity methods saw performance decreases of 20%-35%.
  • Shavarani and Sarkar (2023) and De Cao et al. (2021b) showed the smallest performance drops (around 2%), indicating better adaptability to larger candidate sets.
  • Inference times increased substantially for some models when using the entire in-domain vocabulary as a candidate set (e.g., 90x for Feng et al. (2022), 50x for Kolitsas et al. (2018)).
  • The error analysis revealed a significant increase in incorrect entity predictions when candidate sets were ablated.

Limitations and Future Work

The authors acknowledge that some models could not be included due to the unavailability of their source code or training scripts. They strongly advocate for authors to make their code, trained models, and GERBIL integrations publicly available to facilitate replication and standardization.

Future work could focus on developing entity linking systems that are inherently more robust to the absence of candidate sets, moving beyond the current paradigm of heavy reliance on pre-curated lists. This could involve exploring more sophisticated methods for candidate generation or developing end-to-end systems that can effectively disambiguate across the entire entity vocabulary.

Summary updated

BibTeX
@inproceedings{ong2024unified,
  author = {Nicolas Ong and Hassan Shavarani and Anoop Sarkar},
  title = {{Unified Examination of Entity Linking in Absence of Candidate Sets}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-short.11},
}
Figure