A Unified Multi-Task Learning Framework for Joint Extraction of Entities and Relations

Tianyang Zhao, Zhao Yan, Yunbo Cao, Zhoujun Li

Introduction

Information Extraction (IE) is a fundamental task in natural language processing, aimed at transforming unstructured text into structured knowledge representations, typically in the form of (Subject, Relation, Object) triplets. For instance, extracting ("Baghdad", is_located_in, "Iraq") from a news article.

Historically, IE systems relied on pipelined architectures that first identified entities and then classified their relationships. However, these systems are prone to error propagation, where mistakes in the initial entity recognition phase compound in the relation extraction phase. While recent "joint" models attempt to address this by performing both tasks simultaneously, they often face significant hurdles: a heavy reliance on rigid, manually-crafted question templates for Question Answering (QA) approaches, a dependency on pre-defined Named Entity Recognition (NER) models that struggle with non-standard text spans, and a "closed-world" limitation where models cannot identify relations outside of their training set.

This paper presents a Unified Multi-Task (UMT) framework designed to overcome these limitations through a "relation-middle" extraction strategy ($e_1 \rightarrow r \rightarrow e_2$) and automated question generation.

Methodology

The UMT framework integrates three interdependent sub-tasks into a single architecture, all sharing a common feature encoder. This shared encoder utilizes a BERT layer followed by a "Context Fusion Layer"—consisting of Highway networks and a BiGRU—to capture both local semantics and long-range dependencies across the input sentence.

The UMT framework architecture featuring a shared BERT-based feature encoder and three task-specific heads for subject extraction, relation prediction, and object extraction.

The Three Sub-Tasks

The framework operates in a sequential yet unified manner:

  1. Type-attentional Subject Extraction: Rather than performing general NER, the model uses specific entity types (e.g., "Person", "Organization") as prior signals. This guides the model to locate potential subjects using BIOES tagging, effectively acting as a task-specific attention mechanism.
  2. Subject-aware Relation Prediction (SRP): Once a subject is identified, the model predicts relevant relations. It filters out noise by combining the local features of the subject with the global semantics of the entire sentence. This "global filter" ensures the model only pursues relations that are contextually plausible.
  3. QA-based Object Extraction: To identify the object, the system generates natural language questions. Unlike previous template-based methods, UMT employs a Transformer-based Seq2Seq model to automatically generate diverse questions (e.g., "Where is [Subject] located?") from a seed question. This diversity provides the model with multiple syntactic perspectives, enhancing robustness.
An illustration of the three-step extraction process: identifying the subject based on type, predicting the relation subset, and using generated questions to find the object.

Experimental Evaluation and Results

The framework was rigorously evaluated on standard benchmarks (CoNLL04, ACE05), a large-scale industrial dataset (DuIE2.0), and a specialized dataset for non-predefined relations (Travel20).

  • State-of-the-Art Performance: UMT achieved SOTA results on CoNLL04 and ACE05, outperforming previous QA-based joint models by 3.5% and 1.2% in F1 score, respectively.
  • Zero-Shot and Open-World Extraction: On the Travel20 dataset, which tests the ability to extract relations not seen during training, the model achieved an 88.2 F1 score. This significantly outperformed standard BERT baselines (48.7 F1), demonstrating that natural language question-driven extraction allows the model to handle "fuzzy" queries for novel relations.
  • Ablation Insights: Removing the global relation prediction or the subject-aware filtering led to noticeable performance drops, confirming that the synergy between the sub-tasks is critical to the model's success.

Key Insights

A primary finding of this research is that diversity in QA matters. By asking the same question in multiple ways through automatic generation, the model gains a deeper understanding of the token relationships. Furthermore, using entity types as priors acts as a powerful guide, allowing the model to focus its search space from the very beginning of the extraction process.

Conclusion

The UMT framework represents a significant step forward in joint entity and relation extraction. By moving away from rigid templates and embracing automated question generation within a multi-task structure, it achieves superior accuracy and impressive zero-shot capabilities. Future work will focus on optimizing the efficiency of this "relation-middle" approach for datasets with extremely large relation schemas, where the computational cost of querying every potential relation remains a challenge.

Summary updated

Abstract

Joint extraction of entities and relations focuses on detecting entity pairs and their relations simultaneously with a unified model. Based on the extraction order, previous works mainly solve this task through relation-last, relation-first and relation-middle manner. However, these methods still suffer from the template-dependency, non-entity detection and non-predefined relation prediction problem. To overcome these challenges, in this paper, we propose a unified multi-task learning framework to divide the task into three interacted sub-tasks. Specifically, we first introduce the type-attentional method for subject extraction to provide prior type information explicitly. Then, the subject-aware relation prediction is presented to select useful relations based on the combination of global and local semantics. Third, we propose a question generation based QA method for object extraction to obtain diverse queries automatically. Notably, our method detects subjects or objects without relying on NER models and thus it is capable of dealing with the non-entity scenario. Finally, three sub-tasks are integrated into a unified model through parameter sharing. Extensive experiments demonstrate that the proposed framework outperforms all the baseline methods on two benchmark datasets, and further achieve excellent performance for non-predefined relations.
BibTeX
@inproceedings{zhao2021unified,
  author = {Tianyang Zhao and Zhao Yan and Yunbo Cao and Zhoujun Li},
  title = {{A Unified Multi-Task Learning Framework for Joint Extraction of Entities and Relations}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i16.17707},
}
Figure