Abstract
Mingda Chen, Zewei Chu, Yang Chen, Karl Stratos, Kevin Gimpel. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
2019.
Introduction
Entities—people, places, and concepts—are fundamental units of meaning in Natural Language Processing (NLP). Historically, entities were represented using static embeddings, where a single fixed vector was assigned to a name like "Apple." This approach, however, struggles with polysemy (e.g., "Apple" as a fruit vs. a company) and cannot handle entities not seen during training. While contextualized word representations (CWRs) like BERT and ELMo have addressed some of these issues, there has been a lack of a standardized framework to evaluate how effectively these models capture entity-specific knowledge.
To address this, we introduce EntEval, a comprehensive benchmark suite designed to probe the depth and breadth of entity representations across a variety of downstream tasks.
The EntEval Benchmark
EntEval aggregates several diverse datasets to test different facets of entity understanding, including typing, similarity, and disambiguation. The benchmark includes tasks such as Entity Typing (ET), Coreference Arc Prediction (CAP), and Named Entity Disambiguation (NED).
The benchmark is supported by WIKIENT, a massive training dataset derived from Wikipedia containing approximately 92 million instances and over 3 million unique entities.
Methodology: Context and Descriptions
We evaluate two primary modes of entity representation:
- Contextualized Entity Representations (CER): Encoding an entity based on the specific sentence in which it appears.
- Descriptive Entity Representations (DER): Encoding an entity based on its canonical textual description, such as a Wikipedia lead section.
To enhance these representations, we propose EntELMo, a model trained using Wikipedia's internal hyperlink structure. By leveraging the fact that a mention in a specific context is linked to a specific entity page, the model learns to align context-based encodings with description-based encodings.
Key Insights and Results
Our evaluation reveals several critical findings regarding modern CWRs:
- Contextual Superiority: Dynamically encoded representations (CER) significantly outperform static embeddings, particularly in disambiguation tasks where context is vital for resolving ambiguity.
- BERT vs. ELMo: We observe a distinct behavioral split. ELMo is more robust "out of the box" for zero-shot tasks like entity similarity. In contrast, BERT acts as a "supervised learner," requiring task-specific fine-tuning to reach its full potential. Overall, ELMo achieved the highest average score across the benchmark (58.8) compared to BERT Base (53.5).
- Layer Transferability: In deep models, higher layers (closer to the output) generally prove more useful for EntEval tasks than lower, more lexical layers.
- Knowledge Capacity: BERT Large showed significant gains over BERT Base in complex tasks like Entity Relationship Typing (ERT) and Disambiguation (NED), suggesting that larger model capacity correlates with better storage of world knowledge.
Conclusion
EntEval provides a rigorous framework for assessing how well NLP models "understand" entities. While current contextualized models capture significant entity knowledge, the gap between zero-shot performance and supervised fine-tuning remains a key area for future research. Bridging the gap between context-based and description-based representations will be essential for the next generation of entity-centric AI.
Summary updated