Open Information Extraction from Conjunctive Sentences

Swarnadeep Saha, MausamCOLING 2018 · OpenReview

TL;DR

This paper introduces CALM, a novel coordination analyzer that significantly improves Open Information Extraction (Open IE) by intelligently splitting conjunctive sentences into simpler, more manageable ones. By addressing the common errors of dependency parsers in handling conjunctions, CALM enables Open IE systems to extract substantially more information, boosting yield by up to 1.8x with comparable precision.

What Problem Does This Paper Address?

Open Information Extraction (Open IE) aims to extract structured relational tuples from unstructured text. However, a major bottleneck for existing Open IE systems is their inability to effectively process conjunctive sentences – sentences containing multiple clauses or phrases joined by conjunctions like "and," "or," or "but." Dependency parsers, which are crucial for many Open IE systems, often make errors in identifying the correct boundaries of these conjuncts. This leads to missed extractions or, conversely, the generation of overly complex, unreadable tuples. The paper highlights that this poor handling of conjunctive sentences is a primary reason for significant missed recall in Open IE.

Methods

The core contribution is CALM (Coordination Analyzer using Language Model), a system designed to correct parser errors and split conjunctive sentences. CALM operates in two main phases:

  1. Conjunct Boundary Detection:
    • Rule-based Baseline: A baseline algorithm is developed using dependency parse information (specifically 'cc' and 'conj' edges) to identify conjunct heads and expand them into spans. This baseline is implemented on both Stanford and Clear parsers, with Clear showing better performance.
  2. Language Model Scoring: CALM uses a modified language model score (LMScore) to evaluate the coherence of potential simple sentences formed by splitting conjuncts. This score is designed to penalize ungrammatical boundaries that might otherwise receive high probabilities from standard language models.
  3. Linguistic Constraints: To further refine the process and handle tricky cases, CALM incorporates several linguistic constraints. These include ensuring simple sentences have subjects, not splitting named entities, handling light verbs correctly, avoiding repeated words, and adhering to specific grammatical rules for verb forms.
  4. Hierarchical Coordination Tree (HCTree): For sentences with multiple, potentially nested coordinations, CALM constructs an HCTree. This representation captures the containment relationships between different coordination structures, allowing for a more robust disambiguation of complex conjunct boundaries. This is a key novelty, enabling CALM to enforce consistency across multiple coordination structures.
  5. Simple Sentence Generation:
    • Once CALM identifies the correct conjunct spans, it generates simple sentences. This process is guided by the HCTree, ensuring that distributive coordinations are split while non-distributive ones (indicated by triggers like "between," "among," etc., or organization names) are not.

Datasets

  • British News Corpus (BNC): Used for evaluating CALM's coordination analysis. It contains 577 conjunctive sentences (391 with one conjunction, 186 with multiple).
  • Penn Treebank (PTB): Used for comparison with Ficler's system, specifically for conjunct boundary detection.
  • ClueWeb12 (CW) and Newswire/Wikipedia (NW) datasets: Used for evaluating the downstream Open IE task. 100 conjunctive sentences were randomly sampled from each.

Key Insights and Ideas

  • Conjunctions are a major Open IE bottleneck: The paper strongly emphasizes that the inability to correctly handle conjunctive sentences is a primary cause of missed information in Open IE.
  • Splitting is better than parsing: Instead of trying to perfectly parse complex conjunctive structures, splitting them into simpler sentences is a more effective strategy for downstream tasks like Open IE.
  • Language models can be adapted for coherence: The LMScore modification is crucial for evaluating the quality of simple sentences formed by splitting, going beyond simple word probabilities to assess grammaticality at the split point.
  • Hierarchical structure is key for complex coordinations: The HCTree representation is a novel and effective way to manage and enforce consistency across multiple, nested coordination structures within a single sentence.
  • Linguistic constraints are vital for robustness: Simple language model scores are insufficient; incorporating linguistic rules helps avoid nonsensical splits and preserve important semantic units like named entities.

Key Results

  • Improved Open IE Yield: CALMIE (CALM + Open IE system) achieved up to 1.8x yield compared to state-of-the-art Open IE systems (Open IE 4.2 and ClausIE) on conjunctive sentences.
  • Comparable Precision: The yield improvement came with a moderate increase in precision (e.g., 5-point gain for Open IE 4.2 on CW).
  • Significant gains for multiple conjunctions: CALM showed a nearly 3.2 pt gain for sentences with multiple conjunctions over the Clear parser baseline, highlighting the effectiveness of the HCTree.
  • Competitive Conjunct Boundary Detection: CALM achieved competitive results on conjunct boundary detection compared to the state-of-the-art (Ficler and Goldberg, 2016), with an F1 score of 72.81 vs. 72.7.
  • CALMIE outperforms Ficler's system: On sentences with more than two conjuncts, CALMIE achieved 1.3x better yield than Ficler's system.

Limitations and Future Work

  • Parser Dependency: CALM's implementation is specific to the Clear parser. Minor modifications might be needed for other parsers.
  • Non-Distributive Coordination Handling: While CALM has mechanisms to avoid splitting non-distributive coordinations (e.g., using trigger words), it acknowledges that identifying these can be challenging, especially without explicit semantic understanding.
  • Domain/Genre Specificity: As CALM is not an ML system (beyond its reliance on parsers), its performance might vary across different domains or genres.
  • Ellipsis and Gaping: The paper notes that CALM's splitting approach might not work for sentences with ellipsis and gaping, though these are infrequent in their datasets.
  • Future Work:
    • Improving CALM's ability to identify non-distributive coordinations, potentially using Named Entity Recognition (NER) systems.
  • Embedding CALM's ideas into a machine learning system for potentially better in-domain performance.
  • Using CALM as a constraint within learning systems.
  • Extending CALM to handle ellipsis and gaping.
  • Using CALM to improve parsers themselves.

Summary updated

BibTeX
@inproceedings{saha2018open,
  author = {Swarnadeep Saha and Mausam},
  title = {{Open Information Extraction from Conjunctive Sentences}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2018},
  url = {https://openreview.net/forum?id=tjZqRjsMopV},
}
Figure