TL;DR
This paper introduces a novel approach to entity linking by leveraging Wikipedia categories to model context semantically, moving beyond traditional bag-of-words methods. They also propose an adaptive batch size instance selection strategy to efficiently train models on automatically generated data, achieving state-of-the-art performance on the KBP-10 task.
What Problem Does This Paper Address?
Entity linking, the task of mapping mentions in text to entries in a knowledge base (like Wikipedia), is crucial for knowledge base population. A major challenge is disambiguating mentions, especially when names are ambiguous or have variations. Traditional methods often rely on literal term matching of context words, which can suffer from sparsity and fail to capture deeper semantic relationships. Furthermore, supervised entity linking models require a large amount of labeled training data, which is expensive and time-consuming to create, especially when the knowledge base changes. While automatic data generation can help, the resulting data distribution is often imbalanced, hindering model performance.
Methods
The paper proposes two main advancements:
- Wikipedia-LDA Model: This approach uses Wikipedia categories as "topics" to model the semantic context of mentions and knowledge base entries. It employs a supervised Latent Dirichlet Allocation (LDA) model, specifically "labeled LDA," to learn the posterior distribution of words over Wikipedia categories. This allows for measuring context similarity in a semantic space (based on Wikipedia categories) rather than just a literal term space, mitigating the sparsity issue of traditional methods.
- Batch Size Changing Instance Selection Strategy: To address the problem of imbalanced automatically generated training data, the authors propose an iterative instance selection strategy. This strategy aims to select informative, representative, and diverse batches of instances for training. Crucially, the batch size adapts dynamically at each iteration based on the variance of the classifier's confidence on an un-annotated development set. This flexible batch sizing makes the selection process less sensitive to the initial batch size and improves performance compared to fixed-size batching.
Datasets
- Training and Evaluation: The primary dataset used is the KBP-10 knowledge base and document collection. This collection includes Wikipedia Infoboxes and page text for approximately 818,741 entries. The test data contains 2,250 mentions across Person (PER), Geo-Political Entity (GPE), and Organization (ORG) types, drawn from newswire and blog texts. The training set comprises 3,904 newswire and 1,500 web mentions.
- Auxiliary Data: Wikipedia data (released Oct. 08, 2008) was also used to extract name variants and for training the Wikipedia-LDA model.
Key Insights and Ideas
- Semantic Context is Key: Moving beyond literal term matching for context similarity significantly improves entity linking performance. Wikipedia categories provide a rich semantic space for this.
- Wikipedia Categories as Topics: Using Wikipedia categories as topics in an LDA model effectively captures the semantic meaning of text, enabling better disambiguation.
- Adaptive Batch Sizing for Efficiency: Dynamically adjusting batch size during instance selection is more effective than fixed batch sizes. It allows for faster convergence and better performance by adapting to the learning process.
- Instance Selection Balances Data: The proposed instance selection strategy effectively addresses the distribution imbalance in automatically generated data, leading to more robust models.
- Generic Applicability: The batch size changing method is presented as a generic approach that can benefit active learning for other tasks beyond entity linking.
Key Results
- The Wikipedia-LDA model, when combined with base features, significantly outperforms the baseline system that uses literal term matching for context similarity. For example, the "Base + isa_all" system achieved an overall accuracy of 85.9%, compared to 83.2% for the baseline.
- The instance selection strategy, when applied to auto-generated data ("Auto_Gen+IS"), achieved a significant improvement over using the raw auto-generated data ("Auto_Gen"). The "Auto_Gen+IS" system reached 85.2% accuracy, while "Auto_Gen" achieved 81.2%.
- Combining both the Wikipedia-LDA model and the instance selection strategy ("KBP+Auto_Gen+IS" using KBP-10 training data as initial set) achieved the highest performance on the KBP-10 task, outperforming other top systems in the KBP-10 shared task.
- The flexible batch size scheme consistently outperformed fixed batch sizes across various initial batch sizes, demonstrating its robustness.
Limitations and Future Work
- Category Selection: The paper explores different subsets of Wikipedia categories (All, All-admin, isa_all, isa_class, isa_instance) and finds that
isa_allperforms best. However, the selection of these subsets is based on heuristics and observations, suggesting potential for more automated or refined category selection methods. - Instance Selection Guidance: The instance selection process can be guided by either an un-annotated development set (using confidence) or an annotated development set (using accuracy). While both approaches are explored, the paper notes that a small amount of annotated data (500 articles) can be sufficient for guidance when application sets are unavailable beforehand.
- Future Work: The authors suggest that the batch size changing method is a generic approach that can benefit active learning for other tasks. They also imply that further exploration into category selection and instance selection strategies could yield additional improvements.
Summary updated