Systematic Comparison of Neural Architectures and Training Approaches for Open Information Extraction

Patrick Hohenecker, Frank Mtumbuka, Vid Kocijan, Thomas LukasiewiczEMNLP 2020 · OpenReview

Presented at EMNLP 2020, this research represents a significant leap forward for Open Information Extraction (OIE). By moving away from "black box" model comparisons and focusing on the modular components of neural architectures—and crucially, how they are trained—the authors achieved a staggering 200% performance improvement over previous state-of-the-art results.

Introduction

Open Information Extraction (OIE) is the task of extracting structured triples—(subject, predicate, object)—from unstructured text without being limited to a specific domain or pre-defined set of relations. For example, from the sentence "Sam succeeded in convincing John," an OIE system should extract (Sam, succeeded in convincing, John).

Historically, the field has faced two major hurdles:

  1. The "O-Tag" Imbalance: Most OIE systems use sequence tagging (BIO tags). However, in any given sentence, the vast majority of words are "Outside" (O) the triple. Standard training often makes models "lazy," as they can achieve low loss just by predicting "O" for everything.
  2. Inconsistent Comparisons: Previous research often conflated architectural improvements with different pre-processing or evaluation "tricks," making it difficult to isolate effective components.
Examples of OIE extraction and the corresponding BIO tagging scheme.

Methods: A Modular Approach

The authors adopted a modular "Lego-block" approach to build and test every possible combination of OIE components, categorized into three blocks:

  • Embedding Blocks: Simple word-piece embeddings vs. ALBERT (contextualized representations).
  • Encoding Blocks: BiLSTMs, BiLSTM-CNN hybrids, and Transformers.
  • Prediction Blocks: LSTMs, MLPs, CRFs, and Span-based models.

The most novel contribution is the introduction of four training schemes for the Negative Log-Likelihood (NLL) loss, designed to combat the label imbalance:

  • (a) Standard: Calculate loss on all tags.
  • (b) No-O: Disregard the probability of "Outside" tags.
  • (c) Transitions: Only consider tags where a transition occurs (e.g., from O to Subject-B).
  • (d) Boundaries: Only optimize the first and last tokens of every element in the triple.

Key Results and Insights

The systematic evaluation revealed that the training loss formulation is often more important than the network architecture itself. Scheme (d), which focuses exclusively on the boundaries of extracted facts, emerged as the clear winner across nearly all architectural combinations.

Performance comparison of the four training schemes (a-d), highlighting the superiority of boundary-focused training.

Architectural Synergy

While Transformers are excellent encoders, the authors found that using an LSTM as the prediction block on top of a Transformer encoder yielded the best results. This suggests that the sequential nature of LSTMs remains highly valuable for the final tagging step. Conversely, Conditional Random Fields (CRFs), often considered the gold standard for sequence tagging, performed significantly worse than LSTMs and even MLPs in this context.

Impact of Context

Using ALBERT instead of simple embeddings provided a consistent but relatively small boost (+0.012 $F_1$) compared to the massive gains derived from encoding blocks (+0.063 $F_1$) and optimized training schemes.

Performance Gains

The results were transformative for the OIE16 benchmark. The best-performing model (ALBERT + Transformer + LSTM + Training Scheme d) improved the $F_1$ score by 0.421 and AUC-PR by 0.420. This represents a >200% increase in performance compared to previous state-of-the-art models like SpanOIE or RnnOIE when tested under equal conditions.

Conclusion

This study demonstrates that addressing the inherent label imbalance in OIE through targeted training schemes is more effective than simply increasing model complexity. By focusing on triple boundaries, the researchers achieved a new state-of-the-art that is both more accurate and more robust. Future work may explore extending these systematic training schemes to n-ary relations and fine-tuning contextual embeddings to further push the boundaries of the field.

Summary updated

BibTeX
@inproceedings{hohenecker2020systematic,
  author = {Patrick Hohenecker and Frank Mtumbuka and Vid Kocijan and Thomas Lukasiewicz},
  title = {{Systematic Comparison of Neural Architectures and Training Approaches for Open Information Extraction}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=cFhnuplBeXv},
}
Figure