Progressive Multi-task Learning with Controlled Information Flow for Joint Entity and Relation Extraction

Kai Sun, Richong Zhang, Samuel Mensah, Yongyi Mao, Xudong Liu

Abstract

Joint entity and relation extraction is a fundamental task in information extraction. Traditional methods often struggle with either error propagation in pipeline architectures or insufficient task interaction in basic multi-task learning (MTL) frameworks. We present the Progressive Multitask learning model with Explicit Interactions (PMEI), a framework that facilitates "cross-talk" between tasks via early predictions. By incorporating Stochastic Maps (SM) and Mutual Information (MI) constraints, PMEI effectively controls the flow of information, ensuring that early-stage uncertainties do not degrade final performance while capturing deep inter-task correlations. Our experiments demonstrate state-of-the-art results on major benchmarks, including NYT and WebNLG.

Introduction

In Natural Language Processing (NLP), Entity Recognition (ER) and Relation Classification (RC) are intrinsically linked. For instance, identifying "Paris" and "France" as entities is crucial for recognizing the "capital of" relationship between them. Conversely, knowing a relationship exists can help disambiguate entity types.

Historically, researchers have approached this using two main paradigms:

  1. Pipeline Models: These identify entities first and then classify relations. While simple, they suffer from "error propagation"—mistakes in the first step inevitably derail the second.
  2. Basic Multitask Learning (MTL): These models share a common representation layer (e.g., BiLSTM) but perform classification in isolation. This avoids error propagation but fails to exploit the mutual dependencies between the two tasks.
The Basic Multitask Learning (MTL) model where tasks share a hidden representation but lack explicit interaction.

The PMEI Framework

To address these limitations, we propose PMEI, which treats joint extraction as a progressive process. The architecture is built on three pillars:

1. Early Predictions as Context

PMEI generates intermediate "rough draft" predictions ($Y'$) for both ER and RC. These early guesses serve as additional context for the final classification layers, allowing the model to refine its understanding as it progresses.

2. Explicit Task Interaction

Unlike standard MTL, PMEI enables "cross-talk." The final representation for the Entity task is conditioned on the early predictions of both the Entity and the Relation tasks, and vice versa. This explicit interaction allows the model to capture the inherent correlations between identifying entities and their relationships.

3. Controlled Information Flow via Stochastic Maps

A significant risk of using early predictions is that they may be noisy or incorrect. To prevent these errors from corrupting the final output, PMEI utilizes a Stochastic Map (SM) based on the Variational Information Bottleneck principle. Instead of deterministic layers, we use GRUCells to model the mean and variance of a latent distribution.

The PMEI architecture featuring explicit task interaction and Stochastic Maps (SM) to control information flow.

Furthermore, a Mutual Information (MI) constraint is applied via KL-divergence. This acts as a filter, forcing the model to retain only the most relevant features from the input and early predictions while discarding noise.

Experimental Evaluation

We evaluated PMEI on four standard benchmarks: NYT, WebNLG, NYT10, and NYT11. The model was tested using both static GloVe and pre-trained BERT embeddings to assess its versatility.

Key Results

PMEI achieved state-of-the-art (SOTA) performance across all datasets:

  • NYT: Achieved an F1-score of 90.1% (with BERT), surpassing the previous SOTA (CASREL) of 89.6%.
  • WebNLG: Reached an F1-score of 92.0%, a substantial improvement over the 85.6% baseline MTL.
  • Robustness: The model showed consistent gains in "Exact Match" evaluations, particularly in complex datasets like WebNLG where multi-token entities are common.

Discussion and Insights

Ablation studies confirmed that the "cross-talk" mechanism is a primary driver of performance; removing task interaction led to significant drops in F1-scores. Additionally, PMEI demonstrated a superior "inductive bias." When BERT parameters were frozen, PMEI's margin over standard MTL was even larger than when BERT was fine-tuned, suggesting the architecture itself provides a highly effective structure for learning.

The Stochastic Map's $\beta$ parameter, which regulates information flow, revealed a "sweet spot": too much information introduced noise, while too little prevented the model from leveraging task correlations.

Conclusion

PMEI offers a robust solution for joint entity and relation extraction by combining progressive learning with controlled, explicit task interactions. While capturing multi-token entity boundaries in highly complex datasets remains a challenge, the framework's ability to manage uncertainty and exploit task dependencies sets a new standard for joint NLP tasks. Future work will explore applying this architecture to other domains, such as joint sentiment analysis and aspect extraction.

Summary updated

Abstract

Multitask learning has shown promising performance in learning multiple related tasks simultaneously, and variants of model architectures have been proposed, especially for supervised classification problems. One goal of multitask learning is to extract a good representation that sufficiently captures the relevant part of the input about the output for each learning task. To achieve this objective, in this paper we design a multitask learning architecture based on the observation that correlations exist between outputs of some related tasks (e.g. entity recognition and relation extraction tasks), and they reflect the relevant features that need to be extracted from the input. As outputs are unobserved, our proposed model exploits task predictions in lower layers of the neural model, also referred to as early predictions in this work. But we control the injection of early predictions to ensure that we extract good task-specific representations for classification. We refer to this model as a Progressive Multitask learning model with Explicit Interactions (PMEI). Extensive experiments on multiple benchmark datasets produce state-of-the-art results on the joint entity and relation extraction task.
BibTeX
@inproceedings{sun2021progressive,
  author = {Kai Sun and Richong Zhang and Samuel Mensah and Yongyi Mao and Xudong Liu},
  title = {{Progressive Multi-task Learning with Controlled Information Flow for Joint Entity and Relation Extraction}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i15.17632},
}
Figure