Learning to Branch for Multi-Task Learning

Pengsheng Guo, Chen-Yu Lee, Daniel Ulbricht

Abstract

Training multiple tasks jointly in one deep network yields reduced latency during inference and better performance over the single-task counterpart by sharing certain layers of a network. However, over-sharing a network could erroneously enforce over-generalization, causing negative knowledge transfer across tasks. Prior works rely on human intuition or pre-computed task relatedness scores for ad hoc branching structures. They provide sub-optimal end results and often require huge efforts for the trial-and-error process. In this work, we present an automated multi-task learning algorithm that learns where to share or branch within a network, designing an effective network topology that is directly optimized for multiple objectives across tasks. Specifically, we propose a novel tree-structured design space that casts a tree branching operation as a gumbel-softmax sampling procedure. This enables differentiable network splitting that is end-to-end trainable. We validate the proposed method on controlled synthetic data, CelebA, and Taskonomy.

Introduction

Multi-Task Learning (MTL) has become a cornerstone of efficient deep learning, allowing a single model to perform multiple related tasks simultaneously. By sharing parameters across tasks, MTL models reduce memory footprint and inference time while potentially improving performance through cross-task regularization. However, a significant hurdle remains: determining the optimal architecture. If tasks share too many layers, "negative transfer" can occur, where conflicting gradients from different tasks degrade overall performance. Conversely, insufficient sharing misses out on the benefits of joint training.

The Challenge of Manual Branching

Traditionally, designing an MTL architecture involves manually deciding where the network should "branch" into task-specific paths. This process is often guided by human intuition or expensive pre-computations of task similarity. Given that the number of possible branching configurations grows exponentially with the number of tasks and layers, finding the optimal structure through manual trial and error is practically impossible for complex systems.

Differentiable Branching Strategy

To address this, the authors propose a framework that automates the discovery of the branching structure. The network is modeled as a Directed Acyclic Graph (DAG) where each layer (child) can choose its input from a set of preceding layers (parents).

The differentiable branching operation

The core innovation is the use of Gumbel-Softmax sampling. This technique relaxes the discrete, non-differentiable choice of a branching path into a continuous, differentiable approximation. During training, the model learns both the weights of the operations and the probability of each branching path. As training progresses, a "temperature" parameter is annealed, forcing the model to converge on a discrete tree structure. This allows the entire architecture to be optimized end-to-end using standard backpropagation.

Experimental Results and Structural Discovery

The "LearnToBranch" method was evaluated on synthetic data, the CelebA face attribute dataset, and the Taskonomy dataset for dense vision tasks.

The progression from a dense search space to a learned branching structure

On CelebA, the model automatically discovered intuitive task groupings. For instance, it grouped "Eyeglasses" with "Narrow Eyes" and "Mustache" with "No Beard," demonstrating that it can identify semantic relationships purely from data. On Taskonomy, the model outperformed state-of-the-art baselines like AdaShare and Cross-Stitch. Notably, it achieved superior performance while using significantly fewer parameters than single-task models (e.g., 51M vs. 124M), proving that learned branching is more efficient than human-designed heuristics.

The results on Taskonomy also revealed a hierarchical branching pattern: low-level tasks (like edge detection) branch together early, while high-level tasks (like semantic segmentation) share features deeper into the network before splitting.

Conclusion

The proposed automated branching framework removes the need for manual architecture design in multi-task learning. By making the branching decision differentiable, the model can directly optimize its topology for the specific set of tasks at hand. This approach not only improves performance and efficiency but also provides insights into the underlying relationships between different learning objectives. Future work may involve integrating this method with more general neural architecture search (NAS) to optimize both the branching structure and the specific operations within each layer.

Summary updated

Abstract

Training multiple tasks jointly in one deep network yields reduced latency during inference and better performance over the single-task counterpart by sharing certain layers of a network. However, over-sharing a network could erroneously enforce over-generalization, causing negative knowledge transfer across tasks. Prior works rely on human intuition or pre-computed task relatedness scores for ad hoc branching structures. They provide sub-optimal end results and often require huge efforts for the trial-and-error process. In this work, we present an automated multi-task learning algorithm that learns where to share or branch within a network, designing an effective network topology that is directly optimized for multiple objectives across tasks. Specifically, we propose a novel tree-structured design space that casts a tree branching operation as a gumbel-softmax sampling procedure. This enables differentiable network splitting that is end-to-end trainable. We validate the proposed method on controlled synthetic data, CelebA, and Taskonomy.
BibTeX
@inproceedings{guo2020learning,
  author = {Pengsheng Guo and Chen-Yu Lee and Daniel Ulbricht},
  title = {{Learning to Branch for Multi-Task Learning}},
  booktitle = {International Conference on Machine Learning},
  year = {2020},
  url = {https://openreview.net/forum?id=GaoUw0dThW},
}
Figure