Training multiple tasks jointly in one deep network—known as Multi-Task Learning (MTL)—promises reduced inference latency and better performance compared to training individual models. However, designing the architecture for MTL presents a significant challenge. This paper introduces an automated, differentiable method to design multi-task neural networks by learning exactly where layers should share parameters and where they should "branch" out into task-specific paths.
The "Goldilocks" Problem of MTL
In MTL, the goal is to train a single model to perform several different tasks simultaneously (e.g., identifying faces, detecting edges, and estimating depth). This creates a delicate balancing act:
- Sharing too much: Unrelated tasks might interfere with each other, leading to "negative transfer" where overall performance drops.
- Sharing too little: The model loses the benefits of shared computation, resulting in higher latency and memory usage.
Traditionally, researchers have relied on human intuition or expensive trial-and-error to decide which layers to share. As the number of tasks grows, the possible branching combinations become combinatorially explosive, making manual design nearly impossible.
Methodology: Differentiable Branching
To solve this, the authors propose a differentiable branching algorithm that treats the network structure itself as a learnable parameter.
Tree-Structured Design Space
The network is represented as a Directed Acyclic Graph (DAG) composed of "branching blocks." Each block contains multiple "parent" nodes and "child" nodes.
Differentiable Sampling via Gumbel-Softmax
To make the architecture searchable via backpropagation, the authors utilize the Gumbel-Softmax estimator. During training, each child node "samples" a path from one of the parent nodes. Because Gumbel-Softmax provides a continuous relaxation of a discrete choice, the network can use standard gradient descent to learn which connections are most beneficial for minimizing the total loss.
End-to-End Optimization and Final Selection
Unlike some Neural Architecture Search (NAS) methods that optimize structure and weights separately, this method optimizes both simultaneously. Once training converges, the "soft" probabilities are hardened into a final discrete "tree" structure (selecting the single best parent for each child), and the model is re-trained from scratch to reach peak performance.
Experimental Validation
The researchers validated their approach across three distinct datasets:
- Controlled Synthetic Data: 15 regression tasks to test if the model could correctly group related tasks.
- CelebA: A large-scale face attribute dataset with 40 binary classification tasks (e.g., hair color, wearing glasses).
- Taskonomy: A complex dataset for indoor scene analysis featuring five tasks: semantic segmentation, surface normals, monocular depth, keypoint detection, and edge detection.
Key Insights and Results
- Automated Task Grouping: The model successfully discovered task relatedness without prior human knowledge. In the Taskonomy experiments, it naturally grouped "high-level" geometric tasks (segmentation, depth, normals) together, while branching off "low-level" tasks (edges, keypoints) earlier in the network.
- Superior Performance: On the CelebA dataset, the resulting "LearnToBranch-VGG" model achieved 91.55% accuracy, outperforming the standard Branch-VGG baseline (90.79%) while using fewer parameters (1.94M vs 2.09M). On Taskonomy, the method significantly outperformed AdaShare, a state-of-the-art baseline, reducing the error metric in semantic segmentation from 0.566 to 0.462.
- Efficiency over Complexity: By restricting child nodes to sample only one parent, the network topology remains a tree rather than a dense web. This prevents the search space from growing exponentially and keeps computational costs manageable.
- Robustness: When sampling different architectures after training, the resulting pruned topologies remained highly consistent, indicating the model reliably finds a stable, optimal way to group tasks.
Limitations and Future Work
Like many NAS approaches, this model requires a two-stage process: a search phase followed by a "train from scratch" phase, which increases the total time to deployment. Additionally, while the branching is automated, the number of available nodes in each block remains a user-defined hyperparameter based on computational budgets.
Future directions for this research include extending the framework to handle multi-modality (e.g., combining text and image inputs) and adapting it for datasets with partial annotations, where not every training image has a label for every task.
Summary updated