Abstract
Karan Goel, Laurel Orr, Nazneen Fatema Rajani, Jesse Vig, Christopher Ré. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Papers.
2021.
Introduction
Named Entity Linking (NEL) is the critical task of mapping text mentions to specific knowledge base entries. While large-scale custom systems are standard for tech giants, many organizations rely on "off-the-shelf" (OTS) commercial APIs or academic models. However, these systems often operate as "black boxes" with significant performance biases toward popular entities. This study investigates the performance of OTS systems on "tail" entities—those appearing infrequently—and proposes a "model patching" strategy to repurpose these models for specialized domains, such as sports-themed question answering, with minimal engineering overhead.
Methods and Evaluation Framework
The authors evaluate performance across diverse "subpopulations" using the Robustness Gym tool. This allows for a fine-grained analysis of how models handle very popular "head" entities versus rare "tail" entities, as well as specific linguistic contexts.
To address performance gaps, the paper introduces Model Patching via Data Engineering. Instead of modifying model architecture, the authors utilize weak labeling. This technique involves user-defined heuristics to relabel training data. For example, a rule might identify the phrase "match against" to correctly label a following entity as a "sports team" rather than a "country." The OTS model is then fine-tuned on this patched dataset.
Datasets
Evaluation was conducted on:
- WIKIPEDIA: 100,000 mentions categorized into "head," "torso," "tail," and "toe" based on popularity.
- AIDA (test-b): A standard NEL benchmark.
- Subpopulations: Specific slices like "strong affordance" and "kg-relation" were used to test contextual reasoning.
Key Insights: The Popularity Trap and Academic Superiority
A primary finding is the "Popularity Trap": commercial systems frequently default to the most popular entity candidate. Remarkably, a simple baseline that always selects the most popular entity (POP) outperformed all commercial APIs on the AIDA dataset.
Furthermore, open-source academic models, specifically BOOTLEG, consistently outperformed commercial offerings from Google, Microsoft, and Amazon. This gap was most pronounced on rare entities, where commercial systems lagged by over 10% in recall on the Wikipedia dataset. Commercial systems also showed high sensitivity to surface features like capitalization, with performance dropping by over 20% when mentions were not properly capitalized.
Results: The Power of the Patch
The patching strategy proved highly effective. By applying weak labeling to create BOOTLEGSPORT, the authors achieved a 25% absolute accuracy improvement on mentions where the model previously confused countries with sports teams. Crucially, this improvement was "surgical" — it enhanced domain-specific accuracy without degrading the model's performance on general contexts.
Conclusion
This work demonstrates that OTS NEL systems can be effectively audited and repurposed through data engineering. By treating models as black boxes and focusing on high-quality, heuristically-guided data relabeling, developers can achieve significant performance gains in specialized domains without the need for deep learning expertise or architectural changes. Future research will explore the scalability of this approach across broader sets of domain-specific conflicts.
Summary updated