Abstract
Joint entity and relation extraction is a fundamental task in information extraction. Traditional methods often struggle with either error propagation in pipeline architectures or insufficient task interaction in basic multi-task learning (MTL) frameworks. We present the Progressive Multitask learning model with Explicit Interactions (PMEI), a framework that facilitates "cross-talk" between tasks via early predictions. By incorporating Stochastic Maps (SM) and Mutual Information (MI) constraints, PMEI effectively controls the flow of information, ensuring that early-stage uncertainties do not degrade final performance while capturing deep inter-task correlations. Our experiments demonstrate state-of-the-art results on major benchmarks, including NYT and WebNLG.
Introduction
In Natural Language Processing (NLP), Entity Recognition (ER) and Relation Classification (RC) are intrinsically linked. For instance, identifying "Paris" and "France" as entities is crucial for recognizing the "capital of" relationship between them. Conversely, knowing a relationship exists can help disambiguate entity types.
Historically, researchers have approached this using two main paradigms:
- Pipeline Models: These identify entities first and then classify relations. While simple, they suffer from "error propagation"—mistakes in the first step inevitably derail the second.
- Basic Multitask Learning (MTL): These models share a common representation layer (e.g., BiLSTM) but perform classification in isolation. This avoids error propagation but fails to exploit the mutual dependencies between the two tasks.
The PMEI Framework
To address these limitations, we propose PMEI, which treats joint extraction as a progressive process. The architecture is built on three pillars:
1. Early Predictions as Context
PMEI generates intermediate "rough draft" predictions ($Y'$) for both ER and RC. These early guesses serve as additional context for the final classification layers, allowing the model to refine its understanding as it progresses.
2. Explicit Task Interaction
Unlike standard MTL, PMEI enables "cross-talk." The final representation for the Entity task is conditioned on the early predictions of both the Entity and the Relation tasks, and vice versa. This explicit interaction allows the model to capture the inherent correlations between identifying entities and their relationships.
3. Controlled Information Flow via Stochastic Maps
A significant risk of using early predictions is that they may be noisy or incorrect. To prevent these errors from corrupting the final output, PMEI utilizes a Stochastic Map (SM) based on the Variational Information Bottleneck principle. Instead of deterministic layers, we use GRUCells to model the mean and variance of a latent distribution.
Furthermore, a Mutual Information (MI) constraint is applied via KL-divergence. This acts as a filter, forcing the model to retain only the most relevant features from the input and early predictions while discarding noise.
Experimental Evaluation
We evaluated PMEI on four standard benchmarks: NYT, WebNLG, NYT10, and NYT11. The model was tested using both static GloVe and pre-trained BERT embeddings to assess its versatility.
Key Results
PMEI achieved state-of-the-art (SOTA) performance across all datasets:
- NYT: Achieved an F1-score of 90.1% (with BERT), surpassing the previous SOTA (CASREL) of 89.6%.
- WebNLG: Reached an F1-score of 92.0%, a substantial improvement over the 85.6% baseline MTL.
- Robustness: The model showed consistent gains in "Exact Match" evaluations, particularly in complex datasets like WebNLG where multi-token entities are common.
Discussion and Insights
Ablation studies confirmed that the "cross-talk" mechanism is a primary driver of performance; removing task interaction led to significant drops in F1-scores. Additionally, PMEI demonstrated a superior "inductive bias." When BERT parameters were frozen, PMEI's margin over standard MTL was even larger than when BERT was fine-tuned, suggesting the architecture itself provides a highly effective structure for learning.
The Stochastic Map's $\beta$ parameter, which regulates information flow, revealed a "sweet spot": too much information introduced noise, while too little prevented the model from leveraging task correlations.
Conclusion
PMEI offers a robust solution for joint entity and relation extraction by combining progressive learning with controlled, explicit task interactions. While capturing multi-token entity boundaries in highly complex datasets remains a challenge, the framework's ability to manage uncertainty and exploit task dependencies sets a new standard for joint NLP tasks. Future work will explore applying this architecture to other domains, such as joint sentiment analysis and aspect extraction.
Summary updated