Joint Entity and Relation Extraction with a Hybrid Transformer and Reinforcement Learning Based Model

Ya Xiao, Chengxiang Tan, Zhijie Fan, Qian Xu, Wenye ZhuAAAI 2020 · DOI

Abstract

Extracting structured information from unstructured text is a fundamental task in natural language processing. Traditional methods often struggle with "noisy" labels generated through distant supervision, where external databases are used to automatically annotate text. This paper presents JRE_TRL, a novel hybrid model that jointly extracts entities and relations while filtering out irrelevant or incorrectly labeled data. By integrating a Transformer-based encoder for contextual representation with a Reinforcement Learning (RL) agent for decision-making, the model achieves state-of-the-art performance. Experimental results on the New York Times (NYT) corpus demonstrate that JRE_TRL significantly outperforms existing joint and pipelined models, particularly in its ability to handle noisy training environments.

Introduction

Information extraction (IE) typically involves two sub-tasks: Named Entity Recognition (NER) and Relation Extraction (RE). For instance, given the sentence "Steve Jobs founded Apple," an IE system must identify "Steve Jobs" and "Apple" as entities and "founder_of" as the relationship between them. While early systems used pipelined architectures—performing NER then RE—these often suffered from error propagation. Joint models were developed to address this by learning both tasks simultaneously.

However, a major bottleneck in training these models is the reliance on distant supervision. This technique automatically generates training data by aligning entities in a knowledge base with text. While efficient, it introduces significant noise; not every sentence containing two entities expresses their known relationship. For example, a sentence like "Steve Jobs once ate an Apple in Cupertino" would be incorrectly labeled as evidence for the "founder_of" relation. To address this, we propose JRE_TRL, which utilizes Reinforcement Learning to identify and ignore noisy samples during the joint extraction process.

Methods

The JRE_TRL architecture is composed of four integrated modules designed to capture both local entity features and global sentence context.

  1. Enhanced Embedding Layer: Standard models use token and position embeddings. JRE_TRL adds an attenuation coefficient-based position weight. This "entity-wise position embedding" prioritizes words in close proximity to the target entities, as these words typically carry the most relevant relational cues.
  2. Transformer Encoder: The model leverages a BERT-base Transformer architecture. This allows the system to capture deep, bidirectional contextual information, providing a robust hidden state representation for each token in the input sentence.
  3. LSTM Entity Decoder: Entity detection is treated as a sequence labeling task. An LSTM layer processes the Transformer's output to identify entity boundaries and types (Source, Target, or Other). The resulting entity embeddings are then passed to the relation classifier.
  4. Reinforcement Learning Agent: The RL agent acts as the primary decision-maker for relation classification and noise filtering. It takes the combined state of the Transformer's hidden representations and the LSTM's entity embeddings. The agent then selects an action: either assigning a specific relation type or flagging the sentence as "noise" to be excluded from the training update.
Figure 1: Architecture of the JRE_TRL model, showing the flow from input embeddings through the Transformer and LSTM layers to the RL-based classification.

Training is guided by a reward function that considers the improvement in the model's F1-score and the precision of entity detection. This ensures the RL agent learns to favor "clean" sentences that contribute positively to the extraction performance.

Experimental Setup

The model was evaluated using the New York Times (NYT) dataset, a standard benchmark for distantly supervised relation extraction. The training set consisted of approximately 62,000 sentences. To rigorously test the model, two distinct test sets were used:

  • TestSet1: A manually annotated "clean" set of 370 sentences, used to measure pure extraction accuracy.
  • TestSet2: A randomly sampled "noisy" set of ~3,700 sentences from the original distantly supervised data, used to evaluate the model's robustness to incorrect labels.

Results and Discussion

JRE_TRL demonstrated superior performance across all metrics compared to baseline models like FCM, CoType, and HRL. On the clean TestSet1, the model achieved an entity detection F1-score of 0.725 and a relation extraction F1-score of 0.686.

Figure 4: Comparison of JRE_TRL against baseline methods on TestSet1 (clean) and TestSet2 (noisy). Bold values indicate the best performance.

More importantly, on the noisy TestSet2, JRE_TRL maintained a high relation extraction F1-score of 0.623, significantly outperforming the HRL model (0.606). This gap highlights the effectiveness of the RL-based noise filtering mechanism. Furthermore, the model's ability to correctly identify noisy versus clean sentences was validated through a manual audit of 505 sentences, where the RL agent achieved a precision of 70.58%. This confirms that the agent is not just guessing but is learning meaningful patterns that distinguish valid relational evidence from coincidental entity co-occurrence.

Conclusion

The JRE_TRL model represents a significant step forward in joint entity and relation extraction. By combining the representational power of Transformers with the strategic filtering capabilities of Reinforcement Learning, it effectively mitigates the noise inherent in distantly supervised datasets. While the current iteration focuses on single-triplet sentences, future research will aim to extend this architecture to handle complex scenarios involving overlapping entities and multiple relations within a single sentence.

Summary updated

BibTeX
@inproceedings{xiao2020joint,
  author = {Ya Xiao and Chengxiang Tan and Zhijie Fan and Qian Xu and Wenye Zhu},
  title = {{Joint Entity and Relation Extraction with a Hybrid Transformer and Reinforcement Learning Based Model}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i05.6471},
}
Figure