Abstract
This paper details the University of Illinois Cognitive Computation Group’s (UI-CCG) participation in the 2015 Text Analysis Conference (TAC) Knowledge Base Population (KBP) tracks. The team developed specialized systems for three core tasks: identifying "event nuggets" within text, linking Spanish entities to English Wikipedia, and validating the accuracy of extracted slot fillers. By integrating rich semantic features—such as Semantic Role Labeling (SRL) and Explicit Semantic Analysis (ESA)—with innovative domain adaptation and translation-based strategies, the UI-CCG team achieved top-tier performance, particularly in cross-lingual entity linking.
Introduction
Building structured knowledge bases from unstructured, multi-lingual text remains a significant challenge in Natural Language Processing (NLP). The 2015 TAC KBP tracks provided a platform to evaluate systems on three specific hurdles:
- Event Nugget Detection: Identifying trigger words for events and determining their "Realis" status (actual, hypothetical, or generic).
- Cross-lingual Entity Discovery and Linking (EDL): Identifying Spanish entities and linking them to a central knowledge base like FreeBase or Wikipedia.
- Slot Filler Validation (SFV): Verifying if a proposed fact is supported by a specific document.
The UI-CCG team demonstrated that high performance can be achieved by leveraging mature English-centric tools through translation and adapting models to handle data density mismatches between different datasets.
Cross-lingual Entity Discovery and Linking (EDL)
A standout success for the team was their approach to Spanish EDL. Rather than building a native Spanish pipeline from scratch, which often suffers from a lack of high-quality NLP tools, the team employed a "translation-first" strategy.
As shown in Figure 1, Spanish documents were first translated into English using Google Translate. The mature Illinois Wikifier was then applied to the English text to identify and link entities. Finally, a custom n-gram matching algorithm based on edit distance mapped these English entity mentions back to their original spans in the Spanish text. This approach outperformed many native-language systems, ranking 1st place in both linking and clustering evaluations among all participants.
Event Nugget Detection and Co-reference
The event nugget system utilized a stage-wise supervised approach. A 34-class SVM classifier was used for detection and categorization, employing features such as Named Entity Recognition (NER), Semantic Role Labeling (SRL), and Brown Clusters. A secondary classifier determined the Realis status.
A critical insight during development was the "Density Mismatch" between the ACE2005 and KBP datasets. ACE sentences contain an average of 0.34 events, whereas KBP sentences average 0.82. To bridge this gap, the team used subsampling and resampling techniques to align the event distributions, which significantly boosted recall.
As seen in Figure 4, the best results were achieved in Trial Three, reaching an F1 score of 36.28 for the combined "Type+Realis" metric. This highlights the importance of domain adaptation when training data is scarce or distributed differently across datasets.
Slot Filler Validation (SFV)
The SFV task was framed as a textual entailment problem. The system combined hand-written rules (an "Argument Checker") with machine learning models (Naive Bayes and SVM). The ML component was specifically trained to identify "competing" relations—instances where a new claim might contradict an established fact. This hybrid approach improved the F1 score on 2014 test data from 0.489 to 0.503, demonstrating the value of contextual logic and world knowledge in validation tasks.
Conclusion
The UI-CCG entries for TAC KBP 2015 underscore the effectiveness of leveraging cross-lingual resources and sophisticated semantic features. While the translation-based EDL system was highly successful, the SFV system revealed limitations in handling complex syntactic constraints. Future work will focus on integrating deeper syntactic parsing and more robust world knowledge to improve the precision and recall of fact validation.
Summary updated