Abstract
The CMUML system for the 2015 KBP Cold Start Slot Filling task represents a strategic shift from traditional pattern-matching to "micro-readers"—specialized semantic analysis components designed for sentence-level understanding. By decomposing the reading process into specific linguistic sub-tasks, the system attempts a more granular approach to machine reading. While a last-minute system failure significantly impacted recall during the official evaluation, the results highlight a high-precision capability for deep language understanding.
Introduction
The Cold Start Slot Filling (SF) task requires systems to extract specific information about entities (e.g., a person's birthplace or a company's leadership) from a large document corpus without prior knowledge. Traditionally, systems have relied on pattern detection, which often struggles with the inherent complexity and diversity of natural language. The CMU team proposed a "machine reading" approach, aiming for a holistic understanding of sentence structure and semantics to overcome the limitations of template matching.
System Architecture
The CMUML 2015 system is an ensemble architecture that integrates specialized "micro-readers" with legacy components and validation modules.
The Micro-Readers
The core innovation of the 2015 system is the use of four specialized readers:
- Noun Phrase Categorizer: A logistic regression classifier that predicts fine-grained semantic types for noun phrases based on context.
- Semantic Parsers: Generative and CCG-based parsers that map sentences into formal logic, targeting relations defined in the NELL (Never-Ending Language Learning) project.
- Verb Analyzer: Identifies relations expressed through verbs by aligning them with a predefined ontology.
- Nominal Noun Analyzer: Extracts information from compound nouns and appositives (e.g., identifying "journalist" as a job title from "British journalist John Murray").
Legacy and Validation Components
To ensure baseline performance, the system included a 2014 CRF-based sequence labeler and rule-based extractors. For validation, the system employed OpenEval, a web-based tool that corroborates extracted facts against the live web, and used the ClueWeb/Freebase (FACC) dataset for entity matching and synonym resolution.
Results and Analysis
The official evaluation was impacted by a system failure that prevented legacy components from processing approximately 80% of the queries. Consequently, the reported metrics primarily reflect the performance of the new micro-readers.
The system achieved a respectable precision of 62.5% for the CMUML1 run (CSSF measure). This indicates that the micro-readers are highly accurate when they do identify a relation. However, the recall was extremely low (0.4%), confirming that these specialized components are currently very selective and were unable to compensate for the failure of the broader legacy system. The inclusion of OpenEval in CMUML2 slightly adjusted precision to 60.98% while maintaining similar recall levels.
Conclusion
The CMUML 2015 system demonstrates that decomposing "reading" into specialized linguistic tasks is a viable path toward high-precision information extraction. While the current implementation favors precision over recall, future work will focus on better integrating these deep semantic parsers with high-recall statistical models to achieve a more robust and comprehensive machine reading system.
Summary updated