Dynamic Multi-Task Learning with Convolutional Neural Network

Yuchun Fang, Zhengyan Ma, Zhaoxiang Zhang, Xu-Yao Zhang, Xiang Bai

Abstract

Multi-task learning (MTL) aims to improve model performance by training related tasks together, leveraging shared representations. However, traditional MTL often relies on manual, subjective task grouping, which can lead to "negative transfer" if unrelated tasks are paired. This paper introduces the Dynamic Multi-Task CNN (DMT CNN), a framework that automatically discovers task relationships from data. By utilizing learnable Task Transfer Connections (TTC), the model dynamically adjusts information flow between task-specific subnets. Our evaluation on the CelebA and CIFAR-10 datasets demonstrates that DMT CNN consistently outperforms single-task and traditional multi-task models, while also enabling efficient incremental learning for new tasks.

Introduction: The Challenge of Task Grouping

Multi-Task Learning (MTL) is a powerful paradigm, but it faces two significant hurdles. First, subjective grouping requires researchers to guess which tasks are related. Incorrect groupings can lead to performance degradation. Second, rigid architectures typically use "hard-sharing" (shared initial layers with late branching), which fails to capture the complex, overlapping relationships found in large-scale task sets. The DMT CNN addresses these issues by moving away from hand-crafted groups toward a data-driven approach that learns how much information to share at each layer.

The DMT CNN Architecture

The core of the proposed framework is the Dynamic Multi-Task CNN (DMT CNN). Unlike monolithic backbones, the DMT CNN assigns a dedicated subnet to each task, ensuring task identity and completeness.

The DMT CNN architecture features task-specific subnets connected by learnable Task Transfer Connections (TTC).

The architecture is defined by several key components:

  • Task-Specific Subnets: Each task has its own branch, allowing it to maintain specialized features.
  • Task Transfer Connections (TTC): These are learnable parameters ($\alpha$) that link layers across different subnets. Through gradient descent, the model updates these values to determine the strength of information flow. A high $\alpha$ indicates heavy reliance on another task's features, while an $\alpha$ of zero implies independence.
  • Dynamic Adjustment: The model "softly" partitions tasks into groups by deciding the degree of sharing at every specific layer, rather than imposing a global sharing constraint.

Dynamic Sharing and Task Relatedness

One of the most compelling features of DMT CNN is its ability to visualize and quantify task relatedness. The authors observed that tasks focusing on similar image regions (e.g., "Goatee" and "Sideburns") naturally develop stronger sharing connections.

The sharing degree between tasks evolves dynamically during training, reflecting the learned semantic relationships.

As shown in the training curves, the "sharing degree" for various facial attributes converges to stable values over several epochs. This automated "weak grouping" allows a task to draw varying amounts of information from multiple sources, creating a nuanced web of relationships. Furthermore, analysis revealed that semantic sharing becomes more pronounced in deeper layers, suggesting that while low-level features are universal, high-level cross-talk is critical for task-specific performance.

Experimental Results

The DMT CNN was evaluated on the CelebA dataset (40 facial attributes) and CIFAR-10 (reorganized into five binary tasks).

  • Accuracy Gains: DMT CNN consistently outperformed Single-Task (ST) and traditional Multi-Task (MT) baselines. For instance, on the highly imbalanced "Bald" attribute, DMT achieved 80.1% accuracy, a massive improvement over the 56.4% achieved by a single-task model.
  • Incremental Learning: The architecture supports adding new tasks by creating a new subnet and learning its TTCs to existing, frozen subnets. This process was found to be approximately 4 times faster than training a new model from scratch, as it leverages features already learned by previous tasks.

Conclusion

The Dynamic Multi-Task CNN provides a robust solution to the limitations of manual task grouping in MTL. By learning task relationships through TTCs, it achieves superior accuracy and scalability. While the model currently assumes homogeneous subnet architectures, future work may address the memory overhead of maintaining multiple subnets and explore the integration of heterogeneous structures.

Summary updated

Abstract

Multi-task learning and deep convolutional neural network (CNN) have been successfully used in various fields. This paper considers the integration of CNN and multi-task learning in a novel way to further improve the performance of multiple related tasks. Existing multi-task CNN models usually empirically combine different tasks into a group which is then trained jointly with a strong assumption of model commonality. Furthermore, traditional approaches usually only consider small number of tasks with rigid structure, which is not suitable for large-scale applications. In light of this, we propose a dynamic multi-task CNN model to handle these problems. The proposed model directly learns the task relations from data instead of subjective task grouping. Due to its flexible structure, it supports task-wise incremental training, which is useful for efficient training of massive tasks. Specifically, we add a new task transfer connection (TTC) between the layers of each task. The learned TTC is able to reflect the correlation among different tasks guiding the model dynamically adjusting the multiplexing of the information among different tasks. With the help of TTC, multiple related tasks can further boost the whole performance for each other. Experiments demonstrate that the proposed dynamic multi-task CNN model outperforms traditional approaches.
BibTeX
@inproceedings{fang2017dynamic,
  author = {Yuchun Fang and Zhengyan Ma and Zhaoxiang Zhang and Xu-Yao Zhang and Xiang Bai},
  title = {{Dynamic Multi-Task Learning with Convolutional Neural Network}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2017},
  doi = {10.24963/ijcai.2017/231},
}
Figure