Network Clustering for Multi-task Learning

Dehong Gao, Wenjing Yang, Huiling Zhou, Wei Yi, Yi Hu, Hao Wang

Abstract

The Multi-Task Learning (MTL) technique has been widely studied by word-wide researchers. The majority of current MTL studies adopt the hard parameter sharing structure, where hard layers tend to learn general representations over all tasks and specific layers are prone to learn specific representations for each task. Since the specific layers directly follow the hard layers, the MTL model needs to estimate this direct change (from general to specific) as well. To alleviate this problem, we introduce the novel cluster layer, which groups tasks into clusters during training procedures. In a cluster layer, the tasks in the same cluster are further required to share the same network. By this way, the cluster layer produces the general presentation for the same cluster, while produces relatively specific presentations for different clusters. As transitions the cluster layers are used between the hard layers and the specific layers. The MTL model thus learns general representations to specific representations gradually. We evaluate our model with MTL document classification and the results demonstrate the cluster layer is quite efficient in MTL.

Introduction

In Multi-Task Learning, a single model is trained to perform several different tasks simultaneously, such as classifying sentiment across diverse product categories. Most deep learning approaches currently utilize hard parameter sharing, where a set of "hard layers" is shared by every task to learn general features, followed immediately by "specific layers" unique to each task.

Researchers from Alibaba Group argue that this structure creates a "rapid change" problem. The model is forced to jump directly from a universal representation to a highly specialized one without intermediate steps. This abrupt transition is difficult for the model to estimate effectively and often leads to sub-optimal performance or overfitting.

The NCMTL Framework

To address this limitation, the authors propose Network Clustering Multi-task Learning (NCMTL). The core innovation is the cluster layer, which is placed between the shared hard layers and the task-specific layers to act as a mathematical "buffer."

Structural comparison of MTL before and after network clustering.

Dynamic Clustering and Specialization

During training, the cluster layer dynamically groups tasks with similar weights into $K$ clusters using the K-Means++ algorithm. For all tasks within a single cluster, their individual weights are replaced by the "cluster center" (the average weights of that group). By stacking multiple cluster layers with increasing numbers of clusters (e.g., 3 clusters in layer one, 5 in layer two), the model forces tasks to specialize gradually—moving from "all share" to "groups share" to "individual unique."

Alternating Optimization

The training process alternates between updating the neural network weights via Stochastic Gradient Descent (SGD) and re-clustering the tasks. This ensures that the groupings remain optimal as the model's internal representations evolve.

The dynamic parameter clustering and updating process during training.

Experimental Evaluation

The model was evaluated on 14 Amazon product review datasets (e.g., Electronics, Home/Lifestyle, Media) for binary sentiment classification. The total dataset included 27,755 documents.

Key Results

  • Accuracy: NCMTL achieved an average accuracy of 87.6% across all tasks, outperforming standard HardShare (86.6%) and SoftShare (86.1%) approaches.
  • Semantic Emergence: The model naturally identified shared characteristics between domains, frequently clustering semantically related tasks like "camera," "electronics," and "software" together.
  • Regularization and Efficiency: By forcing similar tasks to share parameters, the cluster layer acts as a form of regularization, preventing the model from memorizing noise. Furthermore, it reduces the parameter size of specific layers by a factor of $N/K$, where $N$ is the number of tasks and $K$ is the number of clusters.

Conclusion

The NCMTL approach demonstrates that learning a smooth transition from general to specific representations significantly improves performance and efficiency in multi-task settings. Future work includes integrating this method with Neural Architecture Search (NAS) to automatically determine the optimal number of clusters and exploring cross-layer clustering architectures.

Summary updated

Abstract

The Multi-Task Learning (MTL) technique has been widely studied by word-wide researchers. The majority of current MTL studies adopt the hard parameter sharing structure, where hard layers tend to learn general representations over all tasks and specific layers are prone to learn specific representations for each task. Since the specific layers directly follow the hard layers, the MTL model needs to estimate this direct change (from general to specific) as well. To alleviate this problem, we introduce the novel cluster layer, which groups tasks into clusters during training procedures. In a cluster layer, the tasks in the same cluster are further required to share the same network. By this way, the cluster layer produces the general presentation for the same cluster, while produces relatively specific presentations for different clusters. As transitions the cluster layers are used between the hard layers and the specific layers. The MTL model thus learns general representations to specific representations gradually. We evaluate our model with MTL document classification and the results demonstrate the cluster layer is quite efficient in MTL.
BibTeX
@misc{gao2021network,
  author = {Dehong Gao and Wenjing Yang and Huiling Zhou and Wei Yi and Yi Hu and Hao Wang},
  title = {{Network Clustering for Multi-task Learning}},
  year = {2021},
  eprint = {2101.09018},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2101.09018},
}
Figure