Bridging Towers of Multi-task Learning with a Gating Mechanism for Aspect-based Sentiment Analysis and Sequential Metaphor Identification

Rui Mao, Xiao LiAAAI 2021 · OpenReview

Abstract

Multi-task learning (MTL) has been widely applied in Natural Language Processing. A major task and its associated auxiliary tasks share the same encoder; hence, an MTL encoder can learn the sharing abstract information between the major and auxiliary tasks. Task-specific towers are then employed upon the sharing encoder to learn task-specific information. Previous works demonstrated that exchanging information between task-specific towers yielded extra gains. This is known as soft-parameter sharing MTL. In this paper, we propose a novel gating mechanism for the bridging of MTL towers. Our method is evaluated based on aspect-based sentiment analysis and sequential metaphor identification tasks. The experiments demonstrate that our method can yield better performance than the baselines on both tasks. Based on the same Transformer backbone, we compare our gating mechanism with other information transformation mechanisms, e.g., cross-stitch, attention and vanilla gating. The experiments show that our method also surpasses these baselines.

Introduction

Multi-Task Learning (MTL) is a cornerstone of modern Natural Language Processing (NLP), designed to improve a model's robustness by training on a primary task alongside related auxiliary tasks. By sharing an encoder, the model learns generalized features beneficial across tasks. However, a significant challenge remains: not all auxiliary information is helpful. Traditional "soft-parameter sharing" techniques, such as cross-stitch networks or attention mechanisms, often fuse information indiscriminately. This can introduce noise or conflicting signals from auxiliary tasks, potentially degrading the performance of the primary task. There is a clear need for a selective mechanism that allows task-specific "towers" to filter and absorb only the most relevant information from their neighbors.

Methodology: The Gated Bridging Mechanism (GBM)

The proposed framework utilizes a shared BERT backbone followed by multiple task-specific towers, each composed of several Transformer layers. The core innovation is the Gated Bridging Mechanism (GBM), which acts as a selective filter between these towers.

The overall architecture of the proposed MTL framework, featuring task-specific towers bridged by the Gated Bridging Mechanism (GBM).

As shown in the architecture overview, the GBM is placed between Transformer blocks of different tasks. It operates through a three-step process inspired by Gated Recurrent Units (GRUs):

  1. Reset Gate: It decides which components of the neighboring tower's hidden states are useful for the current task.
  2. Non-linear Projection: It projects the filtered neighbor information into the current task's vector space using a tanh activation, ensuring alignment between different task representations.
  3. Update Gate: It calculates the optimal trade-off between the tower’s own internal information and the filtered external information.
Detailed view of the Gated Bridging Mechanism (GBM), illustrating the reset and update gates used to selectively fuse information from auxiliary tasks.

Furthermore, the model employs Weighted Sum Pooling to combine features from all Transformer layers. This approach recognizes that different layers capture varying levels of linguistic abstraction—from low-level syntax to high-level semantics—all of which can be critical for complex tasks like metaphor identification.

Experimental Evaluation

The GBM-MTL model was evaluated on two challenging NLP domains: Aspect-based Sentiment Analysis (ABSA), using the Laptop and Restaurant datasets from SemEval-2014, and Sequential Metaphor Identification (SMI), using the VU Amsterdam Metaphor Corpora (VUA).

Key Results

The experiments demonstrated that the GBM-MTL model consistently outperforms strong baselines. On ABSA tasks, it achieved an average gain of 1.26% over the previous state-of-the-art. For metaphor identification, the model reached a 77.6 F1 score, a significant improvement over existing benchmarks.

Ablation studies confirmed the importance of both the GBM and weighted sum pooling. Removing the GBM led to performance drops, highlighting its role in filtering task-specific noise. Probing experiments with random noise further validated the GBM's robustness; it successfully ignored random auxiliary states that would have compromised simpler fusion methods like cross-stitch or attention.

Conclusion

This research introduces a novel gating mechanism that effectively bridges task-specific towers in an MTL framework. By enabling selective knowledge sharing, the GBM addresses the "noise" problem inherent in traditional soft-parameter sharing. The results across sentiment analysis and metaphor identification tasks underscore the effectiveness of this approach. Future work may explore the scalability of GBM to a larger number of tasks and the potential for automated auxiliary task selection.

Summary updated

Abstract

Multi-task learning (MTL) has been widely applied in Natural Language Processing. A major task and its associated auxiliary tasks share the same encoder; hence, an MTL encoder can learn the sharing abstract information between the major and auxiliary tasks. Task-specific towers are then employed upon the sharing encoder to learn task-specific information. Previous works demonstrated that exchanging information between task-specific towers yielded extra gains. This is known as soft-parameter sharing MTL. In this paper, we propose a novel gating mechanism for the bridging of MTL towers. Our method is evaluated based on aspect-based sentiment analysis and sequential metaphor identification tasks. The experiments demonstrate that our method can yield better performance than the baselines on both tasks. Based on the same Transformer backbone, we compare our gating mechanism with other information transformation mechanisms, e.g., cross-stitch, attention and vanilla gating. The experiments show that our method also surpasses these baselines.
BibTeX
@inproceedings{mao2021bridging,
  author = {Rui Mao and Xiao Li},
  title = {{Bridging Towers of Multi-task Learning with a Gating Mechanism for Aspect-based Sentiment Analysis and Sequential Metaphor Identification}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=xJ2xgMbciJM},
}
Figure