Goodwill Hunting: Analyzing and Repurposing Off-the-Shelf Named Entity Linking Systems

Karan Goel, Laurel J. Orr, Nazneen Fatema Rajani, Jesse Vig, Christopher RéNAACL 2021 · OpenReview

Abstract

Karan Goel, Laurel Orr, Nazneen Fatema Rajani, Jesse Vig, Christopher Ré. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Papers.

2021.

Introduction

Named Entity Linking (NEL) is the critical task of mapping text mentions to specific knowledge base entries. While large-scale custom systems are standard for tech giants, many organizations rely on "off-the-shelf" (OTS) commercial APIs or academic models. However, these systems often operate as "black boxes" with significant performance biases toward popular entities. This study investigates the performance of OTS systems on "tail" entities—those appearing infrequently—and proposes a "model patching" strategy to repurpose these models for specialized domains, such as sports-themed question answering, with minimal engineering overhead.

Methods and Evaluation Framework

The authors evaluate performance across diverse "subpopulations" using the Robustness Gym tool. This allows for a fine-grained analysis of how models handle very popular "head" entities versus rare "tail" entities, as well as specific linguistic contexts.

Overview of Off-The-Shelf capabilities and the proposed Model Patching approach using weak labeling.

To address performance gaps, the paper introduces Model Patching via Data Engineering. Instead of modifying model architecture, the authors utilize weak labeling. This technique involves user-defined heuristics to relabel training data. For example, a rule might identify the phrase "match against" to correctly label a following entity as a "sports team" rather than a "country." The OTS model is then fine-tuned on this patched dataset.

Datasets

Evaluation was conducted on:

  • WIKIPEDIA: 100,000 mentions categorized into "head," "torso," "tail," and "toe" based on popularity.
  • AIDA (test-b): A standard NEL benchmark.
  • Subpopulations: Specific slices like "strong affordance" and "kg-relation" were used to test contextual reasoning.

Key Insights: The Popularity Trap and Academic Superiority

A primary finding is the "Popularity Trap": commercial systems frequently default to the most popular entity candidate. Remarkably, a simple baseline that always selects the most popular entity (POP) outperformed all commercial APIs on the AIDA dataset.

Performance comparison across entity popularity slices (head, torso, tail, toes) for commercial and academic models.

Furthermore, open-source academic models, specifically BOOTLEG, consistently outperformed commercial offerings from Google, Microsoft, and Amazon. This gap was most pronounced on rare entities, where commercial systems lagged by over 10% in recall on the Wikipedia dataset. Commercial systems also showed high sensitivity to surface features like capitalization, with performance dropping by over 20% when mentions were not properly capitalized.

Results: The Power of the Patch

The patching strategy proved highly effective. By applying weak labeling to create BOOTLEGSPORT, the authors achieved a 25% absolute accuracy improvement on mentions where the model previously confused countries with sports teams. Crucially, this improvement was "surgical" — it enhanced domain-specific accuracy without degrading the model's performance on general contexts.

Conclusion

This work demonstrates that OTS NEL systems can be effectively audited and repurposed through data engineering. By treating models as black boxes and focusing on high-quality, heuristically-guided data relabeling, developers can achieve significant performance gains in specialized domains without the need for deep learning expertise or architectural changes. Future research will explore the scalability of this approach across broader sets of domain-specific conflicts.

Summary updated

BibTeX
@inproceedings{goel2021goodwill,
  author = {Karan Goel and Laurel J. Orr and Nazneen Fatema Rajani and Jesse Vig and Christopher Ré},
  title = {{Goodwill Hunting: Analyzing and Repurposing Off-the-Shelf Named Entity Linking Systems}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=Hw9qthC7Nv8},
}
Figure