Abstract
Tree-structured multi-task architectures have been employed to jointly tackle multiple vision tasks in the context of multi-task learning (MTL). The major challenge is to determine where to branch out for each task given a backbone model to optimize for both task accuracy and computation efficiency. To address the challenge, this paper proposes a recommender that, given a set of tasks and a convolutional neural network-based backbone model, automatically suggests tree-structured multi-task architectures that could achieve a high task performance while meeting a user-specified computation budget without performing model training. Extensive evaluations on popular MTL benchmarks show that the recommended architectures could achieve competitive task accuracy and computation efficiency compared with state-of-the-art MTL methods. Our tree-structured multi-task model recommender is open-sourced and available at https://github.com/zhanglijun95/TreeMTL.
Introduction
In Multi-Task Learning (MTL), a single model performs several tasks, such as depth estimation and semantic segmentation, simultaneously. While this approach aims to improve efficiency and boost accuracy through shared features, it often faces "negative transfer" or task interference. This occurs when tasks compete for the same neural resources, leading to performance degradation compared to single-task models.
To mitigate this, tree-structured architectures are used, where tasks share initial "root" layers and branch out into task-specific layers. However, determining the optimal branching points for each task is a significant challenge. Manual design requires deep expertise and is often suboptimal, while Neural Architecture Search (NAS) methods are computationally expensive and offer limited control over model constraints. This paper introduces a "Multi-Task Model Recommender" to automate this design process, finding the balance between sharing and branching to optimize performance within specific computational budgets.
Methodology
The proposed framework consists of a three-part system designed to automate the recommendation of efficient tree-structured architectures.
1. Branching Point Detector
Using graph theory (specifically the "Cut Theorem"), this component automatically parses a standard CNN backbone (e.g., ResNet, MobileNet) and identifies logical points where the network can be split into blocks. This transforms a continuous backbone into a set of discrete potential branching locations.
2. Design Space Enumerator
This component employs a recursive algorithm to map out every possible configuration of task sharing and branching across the identified points. This defines the search space of potential multi-task architectures.
3. Task Accuracy Estimator
The core innovation lies in predicting the performance of complex N-task models without exhaustive training. The system trains all possible two-task combinations and estimates the accuracy of a complex model by averaging the performance of its constituent task pairs.
To enhance ranking reliability, the framework utilizes Singular Value Decomposition Entropy (SVDE). This technique measures the stability of a task's performance, allowing the system to down-weight tasks with high variance and ensure the recommended models are robust.
Experimental Evaluation
The framework was evaluated on two major MTL benchmarks: NYUv2 (3 tasks) and Tiny-Taskonomy (5 tasks), using backbones like Deeplab-ResNet34 and MobileNetV2.
Key Results
- Ranking Reliability: The estimated rankings demonstrated a strong correlation with actual "oracle" rankings, with Pearson’s $\gamma$ coefficients ranging from 0.5 to 0.85.
- Performance Gains: On the Taskonomy benchmark, the top-recommended model outperformed the state-of-the-art "What-to-Share" method by 4.8%.
- Efficiency: The recommender identified models that achieved competitive accuracy while reducing FLOPs and parameters by 40-60% compared to independent single-task models.
Discussion
The research highlights several key insights:
- The Power of Pairs: Complex multi-task dynamics can be effectively proxied by pairwise interactions, reducing search time from exponential to manageable.
- White-Box Flexibility: Unlike "black-box" NAS, this recommender provides a performance table, allowing developers to select architectures based on specific FLOP constraints.
- Branching vs. Grouping: Tree-branching architectures prove more efficient than simple task-grouping strategies.
Conclusion
The Multi-Task Model Recommender provides an efficient, automated way to design tree-structured MTL models. While it requires an offline pre-computation phase for two-task pairs, it significantly lowers the barrier to deploying high-performance multi-task systems. Future work may extend this approach to non-vision domains and further explore task relatedness in larger-scale AI systems.
Summary updated