Abstract
We propose a novel progressive parameter-sharing strategy (MPPS) for effectively training multitask learning models on diverse computer vision tasks simultaneously. Specifically, we propose to parameterize distributions for different tasks to control the sharings, based on the concept of Exclusive Capacity that we introduce. A scheduling mechanism following the concept of curriculum learning is also designed to progressively change the sharing strategy to increase the level of sharing during the learning process. We further propose a novel loss function to regularize the optimization of network parameters as well as the sharing probabilities of each neuron for each task. Our approach can be combined with many state-of-the-art multitask learning solutions to achieve better joint task performance. Comprehensive experiments show that it has competitive performance on three challenging datasets (Multi-CIFAR100, NYUv2, and Cityscapes) using various convolution neural network architectures.
Introduction
Multi-Task Learning (MTL) is a powerful paradigm in computer vision, enabling a single neural network to perform multiple tasks—such as semantic segmentation, depth estimation, and surface normal prediction—simultaneously. This approach offers significant advantages in terms of computational efficiency and memory footprint. However, training such models is fraught with challenges.
The primary obstacle is negative transfer, where learning one task interferes with the performance of another. This often occurs when tasks are "competitive," requiring different feature representations that cannot be easily reconciled within a shared set of neurons. Furthermore, designing an optimal sharing architecture—deciding which layers should be shared and which should remain task-specific—is a complex process that typically requires extensive human expertise and trial-and-error.
The authors of this paper argue that the optimization path itself is a critical factor. Forcing a model into a fully shared "hard-parameter sharing" state from the very first epoch can lead to poor local minima, as the optimizer struggles to balance conflicting gradients early in the training process.
Methodology: Progressive Parameter Sharing (MPPS)
To address these issues, the authors introduce Progressive Parameter Sharing (MPPS). Instead of a static, pre-defined architecture, MPPS allows the network's connectivity to evolve dynamically during training through a curriculum learning approach.
Exclusive Capacity (EC)
Central to MPPS is the concept of Exclusive Capacity (EC). EC is defined as the ratio of neurons dedicated exclusively to a single task versus those shared across multiple tasks. A high EC indicates that tasks are largely isolated, while a low EC signifies a high degree of sharing.
Parameterized Masking and Sampling
The mechanism for controlling sharing involves assigning a learnable sharing probability to each neuron in the shared backbone, typically focusing on the higher layers where task-specific semantics are learned. During each training step, binary masks are sampled from these distributions (e.g., Bernoulli or Gaussian). These masks determine which neurons are accessible to a specific task for that iteration.
Curriculum Scheduling
The training process follows a predefined schedule that gradually reduces the EC. The model begins in a state of high isolation, allowing each task to establish stable feature representations without interference. As training progresses, the scheduler forces tasks to share more resources, eventually converging to a standard hard-sharing architecture by the final epoch. This "soft-to-hard" transition acts as a curriculum, simplifying the optimization landscape.
Experimental Results
The MPPS framework was evaluated on three diverse benchmarks: NYUv2 (dense prediction), Cityscapes (urban street-view), and Multi-CIFAR100 (multi-class classification split into 20 sub-tasks).
Performance and Negative Transfer Mitigation
On the NYUv2 dataset, MPPS achieved a 6.5% relative improvement over standard hard-parameter sharing. Notably, it significantly boosted performance on the "Surface Normal" task, which is traditionally prone to negative transfer when paired with segmentation. On Cityscapes, the method yielded a 1.7% improvement.
Integration with State-of-the-Art
One of the key strengths of MPPS is its orthogonality to other MTL techniques. When integrated with methods like MTAN or Cagrad, it consistently improved results, reaching a 10.9% improvement in joint performance when combined with MTAN on NYUv2.
Conclusion and Discussion
MPPS provides a robust way to navigate the trade-offs in multitask learning by treating parameter sharing as a dynamic optimization problem. Because the model eventually converges to a standard shared backbone, it maintains zero inference overhead, making it highly practical for real-world deployment.
While the training phase is more computationally intensive—requiring separate forward and backward passes for each task's sampled masks—the gains in final model accuracy and the reduction in manual architecture design effort are substantial. Future work may explore more efficient sampling methods to scale the approach to even larger numbers of simultaneous tasks.
Summary updated