Introduction
Knowledge Base Population (KBP) involves automatically extracting facts from raw text to populate structured databases. Given that no single Information Extraction (IE) system is perfect, ensembling—combining outputs from multiple systems—is a standard practice. The most effective ensembling method is typically stacking, a supervised approach where a meta-classifier learns to weight systems based on historical performance.
However, stacking faces a "cold start" problem in annual competitions like the NIST TAC KBP. Many participating systems are new or significantly updated, meaning no historical training data exists for them. Relying solely on supervised stacking ignores these innovations, while simple voting schemes often degrade precision. Researchers at UT Austin have developed a hybrid ensembling algorithm that bridges this gap by combining supervised stacking with unsupervised optimization.
Methodology: A Two-Tier Hybrid Architecture
The proposed approach utilizes a two-tier architecture to aggregate outputs from both established and novel systems.
- Unsupervised Tier: For systems lacking historical data, a constrained optimization approach is used. Instead of simple voting, this method aggregates raw confidence scores and calibrates them based on inter-system agreement. This produces a single "Aggregated Confidence" score representing the unsupervised ensemble.
- Supervised Tier: An L1-regularized SVM meta-classifier (the "stacker") then combines the outputs of systems with historical data with the aggregated output from the unsupervised tier.
- Feature Engineering: The meta-classifier incorporates auxiliary features, such as average confidence scores and the degree of system agreement, to refine its final "Accept?" decision.
By treating the unsupervised group as a single "expert" input, the model learns the relative reliability of new systems compared to established ones.
Experimental Setup
The model was evaluated using data from the NIST TAC KBP challenges, specifically using the 2014 competition for training and the 2015 competition for testing. Two primary tasks were targeted:
- Cold Start Slot Filling (CSSF): Extracting attributes (e.g., age, spouse) for specific entities.
- Tri-lingual Entity Discovery and Linking (TEDL): Identifying and linking entities across English, Spanish, and Chinese.
Results and Analysis
The hybrid model established new state-of-the-art benchmarks for both tasks. In the CSSF task, it achieved an F1 score of 0.4489, outperforming the 2015 competition winner and purely supervised baselines.
Key Insights
- Recall Enhancement: The unsupervised tier acted as a significant recall booster. Approximately one-third of the correct facts identified by the final ensemble were unique to the unsupervised systems, highlighting the importance of including new systems that a purely supervised stacker would have ignored.
- The Importance of Calibration: The unsupervised optimization step is critical for normalizing confidence scores across different systems. This ensures that scores from disparate models are meaningfully comparable before being processed by the meta-classifier.
- Robustness: The stacking approach proved more robust for these specific IE tasks than other popular methods, such as Bipartite Graph-based Consensus Maximization (BGCM).
Conclusion
This hybrid approach demonstrates that supervised and unsupervised ensembling methods are complementary. By leveraging historical data where available and intelligent aggregation for new systems, the model significantly improves recall in KBP tasks without sacrificing precision. Future work may focus on reducing the computational overhead of running multiple IE systems and further refining unsupervised confidence calibration.
Summary updated