Deep Multitask Learning with Progressive Parameter Sharing

Haosen Shi, Shen Ren, Tianwei Zhang, Sinno Jialin PanICCV 2023 · OpenReview

Abstract

We propose a novel progressive parameter-sharing strategy (MPPS) for effectively training multitask learning models on diverse computer vision tasks simultaneously. Specifically, we propose to parameterize distributions for different tasks to control the sharings, based on the concept of Exclusive Capacity that we introduce. A scheduling mechanism following the concept of curriculum learning is also designed to progressively change the sharing strategy to increase the level of sharing during the learning process. We further propose a novel loss function to regularize the optimization of network parameters as well as the sharing probabilities of each neuron for each task. Our approach can be combined with many state-of-the-art multitask learning solutions to achieve better joint task performance. Comprehensive experiments show that it has competitive performance on three challenging datasets (Multi-CIFAR100, NYUv2, and Cityscapes) using various convolution neural network architectures.

Introduction

Multi-Task Learning (MTL) is a powerful paradigm in computer vision, enabling a single neural network to perform multiple tasks—such as semantic segmentation, depth estimation, and surface normal prediction—simultaneously. This approach offers significant advantages in terms of computational efficiency and memory footprint. However, training such models is fraught with challenges.

The primary obstacle is negative transfer, where learning one task interferes with the performance of another. This often occurs when tasks are "competitive," requiring different feature representations that cannot be easily reconciled within a shared set of neurons. Furthermore, designing an optimal sharing architecture—deciding which layers should be shared and which should remain task-specific—is a complex process that typically requires extensive human expertise and trial-and-error.

The authors of this paper argue that the optimization path itself is a critical factor. Forcing a model into a fully shared "hard-parameter sharing" state from the very first epoch can lead to poor local minima, as the optimizer struggles to balance conflicting gradients early in the training process.

Methodology: Progressive Parameter Sharing (MPPS)

To address these issues, the authors introduce Progressive Parameter Sharing (MPPS). Instead of a static, pre-defined architecture, MPPS allows the network's connectivity to evolve dynamically during training through a curriculum learning approach.

Exclusive Capacity (EC)

Central to MPPS is the concept of Exclusive Capacity (EC). EC is defined as the ratio of neurons dedicated exclusively to a single task versus those shared across multiple tasks. A high EC indicates that tasks are largely isolated, while a low EC signifies a high degree of sharing.

Figure 1: The progression of Exclusive Capacity from high to low across training epochs. Initially, tasks have dedicated neurons, which gradually transition into a shared pool.

Parameterized Masking and Sampling

The mechanism for controlling sharing involves assigning a learnable sharing probability to each neuron in the shared backbone, typically focusing on the higher layers where task-specific semantics are learned. During each training step, binary masks are sampled from these distributions (e.g., Bernoulli or Gaussian). These masks determine which neurons are accessible to a specific task for that iteration.

Figure 3: Overview of the MPPS architecture. Learnable mask distributions are applied to the higher layers of a shared backbone. Masks are sampled for each task, and gradients are backpropagated to update both the network weights and the mask parameters.

Curriculum Scheduling

The training process follows a predefined schedule that gradually reduces the EC. The model begins in a state of high isolation, allowing each task to establish stable feature representations without interference. As training progresses, the scheduler forces tasks to share more resources, eventually converging to a standard hard-sharing architecture by the final epoch. This "soft-to-hard" transition acts as a curriculum, simplifying the optimization landscape.

Experimental Results

The MPPS framework was evaluated on three diverse benchmarks: NYUv2 (dense prediction), Cityscapes (urban street-view), and Multi-CIFAR100 (multi-class classification split into 20 sub-tasks).

Performance and Negative Transfer Mitigation

On the NYUv2 dataset, MPPS achieved a 6.5% relative improvement over standard hard-parameter sharing. Notably, it significantly boosted performance on the "Surface Normal" task, which is traditionally prone to negative transfer when paired with segmentation. On Cityscapes, the method yielded a 1.7% improvement.

Integration with State-of-the-Art

One of the key strengths of MPPS is its orthogonality to other MTL techniques. When integrated with methods like MTAN or Cagrad, it consistently improved results, reaching a 10.9% improvement in joint performance when combined with MTAN on NYUv2.

Conclusion and Discussion

MPPS provides a robust way to navigate the trade-offs in multitask learning by treating parameter sharing as a dynamic optimization problem. Because the model eventually converges to a standard shared backbone, it maintains zero inference overhead, making it highly practical for real-world deployment.

While the training phase is more computationally intensive—requiring separate forward and backward passes for each task's sampled masks—the gains in final model accuracy and the reduction in manual architecture design effort are substantial. Future work may explore more efficient sampling methods to scale the approach to even larger numbers of simultaneous tasks.

Summary updated

Abstract

We propose a novel progressive parameter-sharing strategy (MPPS) in this paper for effectively training multitask learning models on diverse computer vision tasks simultaneously. Specifically, we propose to parameterize distributions for different tasks to control the sharings, based on the concept of Exclusive Capacity that we introduce. A scheduling mechanism following the concept of curriculum learning is also designed to progressively change the sharing strategy to increase the level of sharing during the learning process. We further propose a novel loss function to regularize the optimization of network parameters as well as the sharing probabilities of each neuron for each task. Our approach can be combined with many state-of-the-art multitask learning solutions to achieve better joint task performance. Comprehensive experiments show that it has competitive performance on three challenging datasets (Multi-CIFAR100, NYUv2, and Cityscapes) using various convolution neural network architectures.
BibTeX
@inproceedings{shi2023deep,
  author = {Haosen Shi and Shen Ren and Tianwei Zhang and Sinno Jialin Pan},
  title = {{Deep Multitask Learning with Progressive Parameter Sharing}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  url = {https://openreview.net/forum?id=vHSjSjCtDT},
}
Figure