Systematic Study of Long Tail Phenomena in Entity Linking

Filip Ilievski, Piek Vossen, Stefan SchlobachCOLING 2018 · OpenReview

Introduction

Entity Linking (EL) is the fundamental process of mapping a mention in text (e.g., "Washington") to a specific entry in a knowledge base like Wikipedia (e.g., George Washington vs. Washington D.C.). While state-of-the-art EL systems report high F1-scores, there is a growing concern that these metrics are inflated by a "head bias." Most evaluation datasets are dominated by famous, frequently mentioned entities, leaving the performance on rare "tail" entities largely unexamined. This paper provides the first systematic investigation into the long tail of EL, quantifying the gap between popular and obscure subjects.

Analytical Framework

The researchers developed a rigorous framework to study the relationship between data properties and system performance across four key dimensions:

  1. Ambiguity: The number of different entities a single name (surface form) can refer to.
  2. Variance: The number of different names that can refer to a single entity.
  3. Frequency: How often a name or entity appears in a corpus.
  4. Popularity: The global importance of an entity, measured via PageRank in DBpedia.

Using this framework, the authors tested 16 hypotheses against three prominent EL systems: AGDISTIS MAG, DBpedia Spotlight, and WAT, utilizing standard benchmarks like AIDA-YAGO2 and the N3 collection.

Log-log distributions of forms, instances, and PageRank across the AIDA and N3 datasets, illustrating the characteristic long-tail distribution of entity data.

The Nature of the Long Tail

The study reveals that entity data follows a Zipfian distribution across multiple dimensions. As shown in the distributions of forms and instances, a small number of "head" entities account for the vast majority of mentions, while a massive "tail" of entities appears only once or twice.

Interestingly, the complexity of an entity often scales with its popularity. The authors found a positive correlation between an entity's PageRank and its variance; more popular entities tend to have a wider variety of names used to refer to them, adding a layer of linguistic complexity to the most frequent cases.

The relationship between an entity's popularity (Log PageRank) and its variance, showing that more prominent entities typically possess a higher number of associated surface forms.

Key Insights: The "Popularity Engine" Trap

A central finding of the research is that most EL systems rely heavily on "statistical priors"—essentially defaulting to the most famous entity associated with a name. While this heuristic succeeds for a basketball star like "Michael Jordan," it fails for a less-famous professor of the same name.

Other critical insights include:

  • Ambiguity is Rare but Deadly: Only 7% of surface forms are ambiguous, but system performance on these cases is 6 to 14 points lower than the average.
  • The Entropy of Confusion: Systems struggle most with "balanced" ambiguity, where two entities are mentioned with similar frequency. When one is dominant, systems simply overfit to the frequent one.
  • Macro vs. Micro F1: Micro F1-scores, which are dominated by frequent entities, often hide the systemic failure of linkers on tail entities. Macro F1-scores reveal a much bleaker picture of current capabilities.

Conclusion and Future Directions

The study concludes that the EL task is far from solved for the long tail. The authors recommend that future research move away from probabilistic priors toward high-precision reasoning that can handle an entity even if it has rarely been seen before. Furthermore, they urge the community to adopt better benchmarks that explicitly include tail statistics and "most-frequent-value" baselines to ensure that high average scores do not mask significant failures on obscure but important subjects.

Summary updated

BibTeX
@inproceedings{ilievski2018systematic,
  author = {Filip Ilievski and Piek Vossen and Stefan Schlobach},
  title = {{Systematic Study of Long Tail Phenomena in Entity Linking}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2018},
  url = {https://openreview.net/forum?id=sJYjxDYID1},
}
Figure