CMUML System for KBP 2015 Cold Start Slot Filling

Bryan Kisiel, Bill McDowell, Matt Gardner, Ndapandula Nakashole, Emmanouil A. Platanios, Abulhair Saparov, Shashank Srivastava, Derry Wijaya, Tom M. Mitchell

Abstract

The CMUML system for the 2015 KBP Cold Start Slot Filling task represents a strategic shift from traditional pattern-matching to "micro-readers"—specialized semantic analysis components designed for sentence-level understanding. By decomposing the reading process into specific linguistic sub-tasks, the system attempts a more granular approach to machine reading. While a last-minute system failure significantly impacted recall during the official evaluation, the results highlight a high-precision capability for deep language understanding.

Introduction

The Cold Start Slot Filling (SF) task requires systems to extract specific information about entities (e.g., a person's birthplace or a company's leadership) from a large document corpus without prior knowledge. Traditionally, systems have relied on pattern detection, which often struggles with the inherent complexity and diversity of natural language. The CMU team proposed a "machine reading" approach, aiming for a holistic understanding of sentence structure and semantics to overcome the limitations of template matching.

System Architecture

The CMUML 2015 system is an ensemble architecture that integrates specialized "micro-readers" with legacy components and validation modules.

The CMUML system architecture, highlighting the integration of micro-readers, legacy extractors, and the OpenEval validator.

The Micro-Readers

The core innovation of the 2015 system is the use of four specialized readers:

  • Noun Phrase Categorizer: A logistic regression classifier that predicts fine-grained semantic types for noun phrases based on context.
  • Semantic Parsers: Generative and CCG-based parsers that map sentences into formal logic, targeting relations defined in the NELL (Never-Ending Language Learning) project.
  • Verb Analyzer: Identifies relations expressed through verbs by aligning them with a predefined ontology.
  • Nominal Noun Analyzer: Extracts information from compound nouns and appositives (e.g., identifying "journalist" as a job title from "British journalist John Murray").

Legacy and Validation Components

To ensure baseline performance, the system included a 2014 CRF-based sequence labeler and rule-based extractors. For validation, the system employed OpenEval, a web-based tool that corroborates extracted facts against the live web, and used the ClueWeb/Freebase (FACC) dataset for entity matching and synonym resolution.

Results and Analysis

The official evaluation was impacted by a system failure that prevented legacy components from processing approximately 80% of the queries. Consequently, the reported metrics primarily reflect the performance of the new micro-readers.

Official performance metrics for the CMUML runs in the KBP 2015 Cold Start Slot Filling task.

The system achieved a respectable precision of 62.5% for the CMUML1 run (CSSF measure). This indicates that the micro-readers are highly accurate when they do identify a relation. However, the recall was extremely low (0.4%), confirming that these specialized components are currently very selective and were unable to compensate for the failure of the broader legacy system. The inclusion of OpenEval in CMUML2 slightly adjusted precision to 60.98% while maintaining similar recall levels.

Conclusion

The CMUML 2015 system demonstrates that decomposing "reading" into specialized linguistic tasks is a viable path toward high-precision information extraction. While the current implementation favors precision over recall, future work will focus on better integrating these deep semantic parsers with high-recall statistical models to achieve a more robust and comprehensive machine reading system.

Summary updated

BibTeX
@inproceedings{kisiel2015cmuml,
  author = {Bryan Kisiel and Bill McDowell and Matt Gardner and Ndapandula Nakashole and Emmanouil A. Platanios and Abulhair Saparov and Shashank Srivastava and Derry Wijaya and Tom M. Mitchell},
  title = {{CMUML System for KBP 2015 Cold Start Slot Filling}},
  booktitle = {TAC},
  year = {2015},
  url = {https://openreview.net/forum?id=DSP8Bq9mRu},
}
Figure