Syntactically Rich Discriminative Training: An Effective Method for Open Information Extraction

Frank Mtumbuka, Thomas LukasiewiczEMNLP 2022 · OpenReview

Introduction

Open Information Extraction (OIE) aims to distill unstructured text into structured (Subject, Relation, Object) tuples. Despite advancements in neural OIE, models often struggle with three key issues: underutilizing syntactic hierarchy, generating repetitive or "hallucinated" text, and failing to extract implicit facts. This paper introduces a novel training framework that addresses these hurdles by leveraging dependency tree structures and a discriminative "real vs. fake" classification task.

Methodology

The proposed architecture, "Syntactically Rich Discriminative Training," employs a multi-pronged strategy to improve extraction quality:

  1. GNN-based Encoding: Rather than treating syntactic tags as flat features, the model uses a Graph Neural Network (GNN) to encode the sentence's dependency tree. A "visibility matrix" ensures words only attend to their direct neighbors in the tree, capturing the semantic skeleton of the sentence.
  2. Discriminative Training: A discriminator module is added to the generator. It classifies each token in a generated tuple as "real" (correct) or "fake" (incorrect), providing a fine-grained signal that forces the model toward higher precision.
  3. Coverage Mechanism: To prevent stuttering, a coverage vector tracks used input parts and penalizes repetitive attention.
  4. Pointer-Generator Networks (PGN): This allows the model to either copy words from the source or generate new ones from a vocabulary, which is crucial for capturing implicit relations.
The proposed framework combining a generator and a discriminator for token-level classification.

Experimental Setup

The model was evaluated on three major benchmarks: OIE2016, CaRB, and LSOIE. A significant contribution of this work is the introduction of paraphrased versions of these datasets. By using the "Parrot" tool to augment training data with diverse lexical forms, the authors improved the model's ability to generalize across different phrasings of the same fact.

Results and Analysis

The integrated approach yielded substantial performance gains. The best configuration—combining ELECTRA embeddings, a GNN encoder, and the discriminator—achieved a 39.63% F1 score improvement over the previous state-of-the-art, IMoJIE, on the CaRB dataset.

Performance comparison of the proposed model against existing OpenIE systems on the CaRB benchmark.

Beyond raw scores, the model effectively addressed common generative failures:

  • Repetition Reduction: The coverage mechanism reduced repetitive tokens by 23% (dropping from 36% to 13%).
  • Implicit Fact Extraction: The PGN allowed the model to introduce relevant vocabulary words not present in the source sentence, forming more coherent facts.

Discussion and Key Insights

The success of this method highlights that the topology of a dependency tree is more valuable than simple tag labels. Respecting the tree structure allows the model to link semantically related words regardless of their linear distance. Furthermore, the discriminator acts as a strict editor, identifying specific errors in extractions that standard loss functions might miss.

While effective, the model currently focuses on binary relations and single-sentence extractions. Future work will need to address n-ary relations and cross-sentence information extraction. Additionally, the "paraphrase gap"—where models trained solely on synthetic data struggle with natural text—remains an area for further research.

Summary updated

BibTeX
@inproceedings{mtumbuka2022syntactically,
  author = {Frank Mtumbuka and Thomas Lukasiewicz},
  title = {{Syntactically Rich Discriminative Training: An Effective Method for Open Information Extraction}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=hQrXdVr37TuJ},
}
Figure