A Semi-shared Hierarchical Joint Model for Sequence Labeling

Gongshen Liu, Du Wei, Jie Zhou, Jing Li, Jie ChengChinese 2023 · DOI

Abstract

Multi-task learning is an essential yet practical mechanism for improving overall performance in various machine learning fields. Owing to the linguistic hierarchy, the hierarchical joint model is a common architecture used in natural language processing. However, in the state-of-the-art hierarchical joint models, higher-level tasks only share bottom layers or latent representations with lower-level tasks thus ignoring correlations between tasks at different levels, i.e., lower-level tasks cannot be instructed by the higher features. This paper investigates how to advance the correlations among various tasks supervised at different layers in an end-to-end hierarchical joint learning model. We propose a semi-shared hierarchical model that contains cross-layer shared modules and layer-specific modules. To fully leverage the mutual information between various tasks at different levels, we design four different dataflows of latent representations between the shared and layer-specific modules. Extensive experiments on CTB-7 and CONLL-2009 show that our semi-shared approach outperforms basic hierarchical joint models on sequence tagging while having much fewer parameters. It inspires us that the proper implementation of the cross-layer sharing mechanism and residual shortcuts is promising to improve the performance of hierarchical joint natural language processing models while reducing the model complexity.

Introduction

Natural language processing (NLP) tasks often follow a natural linguistic hierarchy. For instance, understanding parts of speech (POS) is typically a prerequisite for dependency parsing, which in turn informs deeper semantic role labeling (SRL). Traditional hierarchical multi-task learning (MTL) models mimic this by stacking layers where lower-level tasks are supervised at the bottom and higher-level tasks at the top.

However, these traditional architectures suffer from two primary drawbacks. First, information flow is unidirectional; lower-level tasks do not benefit from the complex semantic features learned at higher levels. Second, model size grows linearly with the number of tasks, leading to parameter explosion. This paper introduces the Semi-shared Hierarchical Joint Model (SSHM) to address these issues by enabling bidirectional information flow and significantly reducing parameter counts.

Methodology: The Semi-shared Mechanism

The core innovation of the SSHM is the separation of parameters into two distinct types of modules:

  1. Layer-Specific Modules ($M_1, M_2, M_3$): Private parameters dedicated to capturing the specific nuances of individual tasks.
  2. Shared Module ($M_0$): A single Transformer-based encoder reused across all levels of the hierarchy.

By reusing $M_0$, the model is forced to learn universal linguistic representations. Gradients from high-level semantic tasks flow back into $M_0$, allowing "top-down" instruction where complex features help refine simpler tasks. The authors explored four methods for combining these modules, as illustrated below.

Four dataflow variants (Straight, Crossed, Semi-combined, and Full-combined) for combining shared ($M_0$) and layer-specific modules.

Among these, Model-IV (Full-combined) proved most effective. In this configuration, outputs from both the previous private and shared modules feed into both the current private and shared modules, maximizing mutual information exchange.

Experimental Results

The SSHM was evaluated on the CTB-7 and CoNLL-2009 datasets, covering a four-task hierarchy: POS tagging, Dependency Parsing (LAS), Predicate Detection (PRE), and Semantic Role Labeling (SRL).

Performance comparison of SSHM variants against the LISA baseline on development and test subsets.

Key Findings

  • Superior Performance: Model-IV consistently outperformed the state-of-the-art LISA model. On the CoNLL-2009 test set, it achieved a Macro F1 of 78.24% compared to LISA's 76.70%.
  • Massive Compression: Despite the better performance, the SSHM (with an 8/8 shared head ratio) contains only 46% of the parameters of the LISA model. This demonstrates that reusing the same module across different depths acts as a powerful regularizer while saving space.
  • SRL and POS Gains: The most complex task, Semantic Role Labeling, saw significant improvements, while lower-level tasks like POS tagging also benefited from the "top-down" influence of shared semantic features.

Conclusion

The Semi-shared Hierarchical Joint Model demonstrates that parameter reuse in hierarchical NLP tasks is not just an efficiency gain but a performance booster. By allowing higher-level features to "instruct" lower-level modules through a shared encoder, the model achieves a more holistic understanding of language. This approach offers a promising path for improving hierarchical joint models while reducing model complexity for deployment.

Summary updated

Abstract

Multi-task learning is an essential yet practical mechanism for improving overall performance in various machine learning fields. Owing to the linguistic hierarchy, the hierarchical joint model is a common architecture used in natural language processing. However, in the state-of-the-art hierarchical joint models, higher-level tasks only share bottom layers or latent representations with lower-level tasks thus ignoring correlations between tasks at different levels, i.e., lower-level tasks cannot be instructed by the higher features. This paper investigates how to advance the correlations among various tasks supervised at different layers in an end-to-end hierarchical joint learning model. We propose a semi-shared hierarchical model that contains cross-layer shared modules and layer-specific modules. To fully leverage the mutual information between various tasks at different levels, we design four different dataflows of latent representations between the shared and layer-specific modules. Extensive experiments on CTB-7 and CONLL-2009 show that our semi-shared approach outperforms basic hierarchical joint models on sequence tagging while having much fewer parameters. It inspires us that the proper implementation of the cross-layer sharing mechanism and residual shortcuts is promising to improve the performance of hierarchical joint natural language processing models while reducing the model complexity.
BibTeX
@inproceedings{liu2023semi,
  author = {Gongshen Liu and Du Wei and Jie Zhou and Jing Li and Jie Cheng},
  title = {{A Semi-shared Hierarchical Joint Model for Sequence Labeling}},
  booktitle = {Chinese},
  year = {2023},
  doi = {10.23919/cje.2020.00.363},
}
Figure