Abstract
This paper describes the JHU/UR pipeline developed for the 2018 NIST SM-KBP evaluation. The system adopts a "translate-then-extract" strategy to address the challenges of Multilingual Knowledge Base Population (KBP) in English, Russian, and Ukrainian. By converting non-English documents into English and applying a suite of neural Information Extraction (IE) tools—including biLSTM-CRF taggers for entities and Siamese networks for name similarity—the pipeline establishes a robust baseline for populating knowledge bases with entities, relations, and events across multiple languages.
Introduction
The goal of the NIST SM-KBP evaluation is to extract structured information from large, unstructured corpora. This involves identifying entities (people, organizations, locations), the relations between them, and the events in which they participate. The 2018 task introduced significant complexities, including cross-lingual requirements (Russian and Ukrainian), disjointed evidence across sentences, and "Task 1b," which requires systems to condition their extraction on external hypotheses or partial knowledge fragments.
The JHU/UR system addresses these challenges through a modular pipeline that prioritizes mature English-language IE tools by first translating all source material into a common English representation.
System Architecture and Methodology
Machine Translation and Alignment
Rather than developing native extractors for Russian and Ukrainian, the system utilizes OpenNMT (a 4-layer bi-directional LSTM) to translate all non-English text into English. To ensure the extracted information can be traced back to the original source—a requirement for the evaluation—the team employed fast-align, an HMM-based aligner. Interestingly, the authors found that traditional HMM-based hard alignments were more reliable for mapping spans back to source documents than the soft attention weights typically produced by neural MT models.
Information Extraction (IE) Components
Once translated, the English text is processed through several specialized modules:
- Entity Mention and Type Detection: The system uses a biLSTM-CRF tagger with GloVe embeddings. It employs a "BIOE" (Begin, Inside, Outside, End) tagging scheme to jointly determine entity boundaries and types.
- Relation and Event Detection: The authors observed that most sentences contain at most one relation mention. Consequently, they treated detection as a sentence-level binary classification task. The model uses an encoder that combines mean-pooling and max-pooling over word embeddings to identify the presence of specific relation or event types.
- Argument Linking: After an event is detected, a separate biLSTM-CRF model identifies which entities within the sentence serve as arguments for that event (e.g., identifying the "attacker" or "target" in a conflict event).
Hypothesis Conditioning (Task 1b)
To handle Task 1b, where the system must adjust its output based on external hypotheses, the pipeline incorporates two specific techniques:
- Siamese Networks: A dual bi-directional LSTM architecture calculates the cosine similarity between entity strings in the document and those provided in the hypothesis.
- Forced Decoding: This is a key innovation where the system manually adjusts the weights of the 'O' (Outside) tag during the CRF decoding process. By lowering the weight of the 'O' tag for entities suggested by the hypothesis, the system artificially boosts recall, ensuring that salient information requested by the user is not filtered out by the standard model thresholds.
Datasets and Training
The pipeline was trained using a diverse array of resources:
- MT Training: A combination of WMT’17 news data, Common Crawl, Yandex, Wiki Headlines, the UN Parallel Corpus, and OpenSubtitles 2018.
- IE Training: Annotated documents from the DARPA AIDA and DEFT programs (specifically ERE data).
- Name Similarity: A dataset derived from WikiNames.
- Evaluation: The 2018 NIST SM-KBP pilot dataset.
Results and Discussion
The system was designed to serve as a high-performing baseline for the pilot task. Key performance metrics include:
- Entity Detection: Achieved an F1 score of 59.5% (83.6% accuracy).
- Argument Linking: The system reached an F1 of 34.8% for relational arguments and 46.8% for event arguments.
- Task 1b Performance: The forced decoding strategy successfully improved recall for hypothesized entities without a catastrophic loss in precision, provided the entities were actually present in the text.
Limitations and Future Work
As a pipeline-based system, the JHU/UR model suffers from error propagation. If the Machine Translation stage mistranslates a name or a verb, the downstream IE components cannot recover that information. Additionally, the system struggled with data sparsity for rare relation and event types, as the models were unable to make predictions for types with no labeled training instances.
Future work will focus on moving toward end-to-end neural architectures that can jointly optimize translation and extraction, thereby reducing the impact of cascading errors and better handling the stylistic variations between newswire and social media data.
Summary updated