Reinforcement Learning-based N-ary Cross-Sentence Relation Extraction

Chenhan Yuan, Ryan A. Rossi, Andrew Katz, Hoda EldardiryarXiv 2020 · OpenReview

TL;DR

This paper tackles the challenge of n-ary cross-sentence relation extraction, where relationships involve multiple entities and can span across non-consecutive sentences. They introduce a novel reinforcement learning-based approach to filter noisy data from distant supervision and a universal relation extractor that handles both consecutive and non-consecutive sentences, achieving state-of-the-art results.

What Problem Does This Paper Address?

Extracting relationships between entities is crucial for building knowledge graphs. While intra-sentence, binary relation extraction is well-studied, n-ary cross-sentence relation extraction, where a single relation involves n entities and can be described across multiple sentences, is more complex. A common approach, distant supervision, assumes that if a group of consecutive sentences contains entities known to have a relation in a knowledge base, then those sentences describe that relation. However, this assumption has two major drawbacks:

  1. Noisy Labels: It often leads to incorrectly labeled sentences, which degrade model performance.
  2. Limited Scope: It fails to capture relations described by non-consecutive sentences, reducing the model's generalizability.

This paper aims to overcome these limitations by proposing a method that can handle noisy data and extract relations from both consecutive and non-consecutive sentences.

Methods

The proposed solution consists of two main components:

  1. Sentence Distribution Estimator (SDE): This is a novel two-level reinforcement learning (RL) agent designed to identify and select correctly labeled sentence groups, thereby mitigating the impact of noisy data.
  2. Main Policy: Selects the primary sentence that likely contains the core information for the relation.
  3. Supplementary Policy: Selects additional sentences that provide supporting information, using indicators like sentence distance to the main sentence, entity variety, and semantic similarity. This policy uses a Monte Carlo search to estimate rewards for intermediate states.
  4. Universal Relation Extractor (RE): This model is designed to encode and extract relations from sentence groups, regardless of whether they are consecutive or not. It employs a hybrid approach:
  5. Bidirectional-LSTM (Bi-LSTM): Encodes each sentence using word embeddings and position encoding.
  6. Piece-wise Convolutional Neural Network (PCNN): Extracts features from the Bi-LSTM output, considering entity positions.
  7. Attention and Self-Attention Mechanisms: Captures dependencies and context across sentences, crucial for handling non-consecutive sentence groups.
  8. Gate Layer and Non-linear Transformation: Integrates information from attention and PCNN outputs.

The SDE and RE models are trained iteratively, with the SDE learning to provide cleaner data for the RE, and the RE's performance feeding back into the SDE's reward signal.

Datasets

The paper evaluates its approach on two datasets:

  • PubMed: A dataset automatically labeled from biomedical literature. This dataset primarily uses the strong distant supervision assumption, focusing on consecutive sentences.
  • WikiText: A new dataset created using a weaker distant supervision assumption. It collects Wikipedia pages and uses Wikidata to label facts, allowing for relations to be described by non-consecutive sentences. This dataset is designed to test the model's ability to handle both consecutive and non-consecutive sentence groups.

Key Insights and Ideas

  • Reinforcement Learning for Noise Filtering: Applying RL to select high-quality sentence groups is an effective way to combat the inherent noise in distant supervision for n-ary relation extraction.
  • Weaker Distant Supervision: Relaxing the "consecutive sentences" assumption to include sentences with at least one main entity or two supplementary entities significantly broadens the scope of extractable relations.
  • Universal Relation Extractor: Combining attention mechanisms with PCNN and Bi-LSTM creates a flexible model capable of handling diverse sentence structures and contexts, making it applicable to both consecutive and non-consecutive sentence scenarios.
  • Importance of Contextual Indicators: The supplementary policy's use of indicators like entity variety and semantic similarity is crucial for effectively selecting relevant non-consecutive sentences.

Key Results

  • PubMed: The proposed RE model achieved higher accuracy than baseline models on most ternary relation tasks. When combined with the SDE, the model further improved performance, demonstrating the effectiveness of noise reduction. The model maintained high accuracy on multi-class tasks, outperforming SotA by 1.8%.
  • WikiText: The RE model alone showed a 7.1% improvement over baselines on this dataset, highlighting its ability to encode non-consecutive sentences. The full model (RE with SDE) achieved an accuracy of 66.4%, significantly outperforming other models and indicating the SDE's role in learning the true relation distribution.
  • SDE Evaluation: An analysis of the SDE's probability distribution showed a strong correlation with human evaluation, confirming its ability to distinguish correctly labeled sentences from noisy ones.

Limitations and Future Work

  • Binary Classification Simplification: The paper notes that for simplification, they used binary classification for the RE model's loss function, though the case is the same for multi-class classification.
  • PCNN Limitations: The paper acknowledges that the PCNN layer might not be ideal for sentences with only one entity in binary relation extraction tasks, as it requires at least two entities for its piecewise pooling.
  • Future Work: The paper suggests that the iterative training of SDE and RE models is key to their success. Further theoretical analysis of the convergence properties and the impact of noise is mentioned as ongoing work. The proposed universal relation extractor is presented as a foundation for future work in various cross-sentence relation extraction scenarios.

Summary updated

Abstract

The models of n-ary cross sentence relation extraction based on distant supervision assume that consecutive sentences mentioning n entities describe the relation of these n entities. However, on one hand, this assumption introduces noisy labeled data and harms the models' performance. On the other hand, some non-consecutive sentences also describe one relation and these sentences cannot be labeled under this assumption. In this paper, we relax this strong assumption by a weaker distant supervision assumption to address the second issue and propose a novel sentence distribution estimator model to address the first problem. This estimator selects correctly labeled sentences to alleviate the effect of noisy data is a two-level agent reinforcement learning model. In addition, a novel universal relation extractor with a hybrid approach of attention mechanism and PCNN is proposed such that it can be deployed in any tasks, including consecutive and nonconsecutive sentences. Experiments demonstrate that the proposed model can reduce the impact of noisy data and achieve better performance on general n-ary cross sentence relation extraction task compared to baseline models.
BibTeX
@misc{yuan2020reinforcement,
  author = {Chenhan Yuan and Ryan A. Rossi and Andrew Katz and Hoda Eldardiry},
  title = {{Reinforcement Learning-based N-ary Cross-Sentence Relation Extraction}},
  year = {2020},
  eprint = {2009.12683},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2009.12683},
}
Figure