On the Surprising Effectiveness of Name Matching Alone in Autoregressive Entity Linking

Elliot Schumacher, James Mayfield, Mark DredzeACL 2023

TL;DR

This paper investigates why the autoregressive language model GENRE achieves state-of-the-art results in entity linking despite ignoring knowledge base (KB) information. They find GENRE heavily relies on memorizing name patterns from its training data, struggling with generalization. The authors then explore augmenting GENRE with KB information (GENRE-KB) to improve its performance on unseen entities and different domains.

What Problem Does This Paper Address?

Entity linking (EL) is the task of identifying mentions of entities in text and linking them to their corresponding entries in a knowledge base (KB). For years, EL systems have relied on a variety of information sources: the similarity between the mention and the entity name, the contextual relevance of the mention within the text, and features from the KB itself (like entity types or popularity). However, a recent autoregressive language model called GENRE (De Cao et al., 2021) achieved impressive results by only using the mention and its surrounding text, completely disregarding KB information. This raises a fundamental question: how can a system that ignores rich KB data outperform those that leverage it? The authors aim to understand GENRE's surprising effectiveness and explore whether incorporating KB information can further improve its capabilities, especially for generalization to new entities and domains.

Methods

The core of the paper revolves around analyzing GENRE and proposing an extension, GENRE-KB.

  • GENRE Analysis: The authors probe GENRE's behavior by:
  • Evaluating its performance on new datasets and KBs to assess generalization.
  • Removing context and the mention string independently to understand the reliance on name matching versus context.
  • GENRE-KB Development: To address GENRE's limitations, the authors propose GENRE-KB. This involves:
  • Augmenting the training data by appending keywords extracted from the KB entity descriptions to the target entity names.
  • Training GENRE to generate both the entity title and these keywords.
  • Using an unchanged GENRE architecture but modifying the training data to incorporate KB information.
  • Keyword selection is done using the PKE toolkit, specifically the Topic Rank algorithm, with a frequency factor. For training, keywords are sampled to avoid memorization, while for inference, the top-scoring keywords are used.

Datasets

The paper utilizes several datasets for training and evaluation:

  • Wikipedia: The primary dataset used for training GENRE-KB, derived from the BLINK-created version of a Wikipedia dataset (Wu et al., 2020) containing 5.9 million entities and 9 million Wikipedia-linked mentions.
  • TAC KBP: The 2015 TAC KBP Entity Linking dataset, consisting of newswire and discussion forum posts. This dataset is challenging due to informal entity mentions.
  • Wikia: A dataset constructed from the Wikia.com website (now Fandom), comprising community-written encyclopedias. This dataset is used to evaluate GENRE and GENRE-KB on domains and KBs different from Wikipedia.

Key Insights and Ideas

  • GENRE's Success is Memorization: The most striking insight is that GENRE's strong performance, particularly on Wikipedia, is largely due to its ability to memorize name patterns from its massive training data. It excels when entity names are lexically similar to mentions but struggles to generalize to unseen entities or when context is crucial.
  • Name Matching Dominates: Experiments show that GENRE relies heavily on name matching. Removing context significantly degrades performance, while removing the mention string also causes a substantial drop, indicating that name patterns are the primary driver of its success.
  • KB Information is Still Valuable: Despite GENRE's success without it, the paper demonstrates that incorporating KB information (via keywords) into GENRE-KB can improve performance, especially on more challenging datasets and when entities are not easily disambiguated by name alone.
  • Generalization is a Major Hurdle: The study highlights the inherent difficulty of achieving true generalization for autoregressive models in entity linking, especially when they are trained on data that encourages memorization.
  • Context Matters for Harder Cases: While GENRE excels at lexically similar matches, GENRE-KB shows an advantage in scenarios where context is more critical for disambiguation, suggesting that KB information can help bridge these gaps.

Key Results

  • GENRE Performance: On the Wikipedia dataset, GENRE achieved high accuracy (e.g., 92.1% micro accuracy on TAC). However, its performance dropped significantly on unseen entities (50% accuracy) and when context was removed.
  • GENRE-KB Improvements: GENRE-KB showed improvements over GENRE on the TAC dataset, particularly in the "oracle" setting (where the gold standard entity is added to the candidate set), indicating its benefit for harder-to-link entities. On Wikia, GENRE-KB performed better on validation sets, and while differences were smaller on test sets, it showed promise for cross-domain linking.
  • Lexical Similarity is Key: For Wikipedia, a high percentage of examples (82.9%) had a Jaro-Winkler score of 0.8 or higher, explaining GENRE's strong performance due to lexical similarity.
  • Model Combination: Combining GENRE and GENRE-KB predictions (using a simple metric like Jaro-Winkler similarity to weigh their outputs) outperformed both individual models on some datasets, suggesting a synergistic effect.

Limitations and Future Work

  • Limitations:
    • Memorization vs. Generalization: The primary limitation is GENRE's reliance on memorization, hindering its ability to generalize to new entities and KBs.
  • Data Skew: The training data is skewed towards specific match types (e.g., exact matches), which may not reflect real-world scenarios.
  • Computational Resources: Training large models from scratch is computationally expensive, limiting the scope of experiments.
  • English-Centric: The experiments are limited to English-language entity linking.
  • Future Work:
    • Targeted Keyword Selection: Developing more sophisticated keyword selection methods that specifically aim to differentiate entities could improve performance.
  • Training from Scratch: Training models from scratch rather than fine-tuning checkpoints could lead to better performance.
  • Novel Architectures: Exploring autoregressive models with novel architectures that can better integrate and learn representations of entities and KB information.
  • Multilingual Linking: Extending the approach to multilingual entity linking.

Summary updated

BibTeX
@inproceedings{schumacher2023surprising,
  author = {Elliot Schumacher and James Mayfield and Mark Dredze},
  title = {{On the Surprising Effectiveness of Name Matching Alone in Autoregressive Entity Linking}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure