Introduction
The Knowledge Base Population (KBP) task of Slot Filling involves automatically extracting specific attributes (slots) for target entities from large-scale unstructured text. Researchers from LMU Munich participated in the TAC 2015 evaluation with a system that integrates deep learning—specifically Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs)—alongside traditional pattern matching and robust coreference resolution. Their hybrid approach secured 3rd place in the competition, highlighting the efficacy of neural models even when trained on noisy, distantly supervised data.
System Architecture
The authors developed a modular pipeline to handle the complexities of natural language diversity and entity ambiguity. The process begins with query processing and document retrieval, followed by sophisticated entity linking and candidate extraction.
Key components of this pipeline include:
- Information Retrieval & Entity Linking: Using the Terrier IR platform and the WAT linker to ensure retrieved documents are relevant to the specific query entity.
- Candidate Extraction: Utilizing Stanford CoreNLP for sentence and filler extraction, enhanced with custom heuristics for coreference resolution to capture mentions like "he" or "the company."
- Relation Classification: A hybrid scoring system combining patterns, SVMs, and neural networks.
Neural Relation Classification
A major innovation in this entry was the deployment of neural networks for relation classification. To overcome the limitations of hand-crafted patterns, the team implemented:
- CNNs: These models split sentences into left, middle, and right contexts relative to the entity and slot filler, extracting local n-gram features.
- RNNs: Various architectures were tested, including forward, bi-directional, and multi-task RNNs that predict the next word's type to better model linguistic structure.
Because training data was generated via "distant supervision" (mapping Freebase facts to text), it contained significant noise. The team employed an iterative filtering process where an SVM trained on a small set of clean data helped select the most reliable samples from the noisy pool for neural training.
Datasets and Evaluation
The system was trained on a massive multi-source corpus including the TAC source corpus, the New York Times corpus, Wikipedia, and Freebase descriptions. Evaluation was conducted on the TAC Cold Start 2015 dataset. To facilitate development, the researchers created a custom set by automatically aligning previous years' manual labels with their source sentences.
Key Results
The CIS system's performance demonstrated the significant impact of both neural architectures and coreference resolution.
As shown in the results table, Run 3 achieved the highest overall F1 score of 21.21. Key takeaways from the evaluation include:
- Neural Boost: Integrating CNNs and RNNs provided a substantial F1 increase of approximately 6 points over a baseline using only patterns and SVMs.
- Coreference Necessity: Disabling coreference resolution led to a 5-point drop in F1, proving it is essential for capturing "inverse slots" and pronominal references.
- Hybrid Success: Combining different models allowed the system to leverage the high precision of patterns while benefiting from the generalization capabilities of neural networks.
Limitations and Future Work
Despite its success, the system faced challenges with data sparsity for certain slots (e.g., per:charges), requiring a fallback to simple patterns. Additionally, while entity linking improved precision, it occasionally reduced recall by incorrectly filtering out valid documents. Future research will focus on improving the quality of distantly supervised training data and refining the ensemble methods used to combine model scores.
Summary updated