Abstract
Multi-task learning (MTL) aims to improve model performance by training related tasks together, leveraging shared representations. However, traditional MTL often relies on manual, subjective task grouping, which can lead to "negative transfer" if unrelated tasks are paired. This paper introduces the Dynamic Multi-Task CNN (DMT CNN), a framework that automatically discovers task relationships from data. By utilizing learnable Task Transfer Connections (TTC), the model dynamically adjusts information flow between task-specific subnets. Our evaluation on the CelebA and CIFAR-10 datasets demonstrates that DMT CNN consistently outperforms single-task and traditional multi-task models, while also enabling efficient incremental learning for new tasks.
Introduction: The Challenge of Task Grouping
Multi-Task Learning (MTL) is a powerful paradigm, but it faces two significant hurdles. First, subjective grouping requires researchers to guess which tasks are related. Incorrect groupings can lead to performance degradation. Second, rigid architectures typically use "hard-sharing" (shared initial layers with late branching), which fails to capture the complex, overlapping relationships found in large-scale task sets. The DMT CNN addresses these issues by moving away from hand-crafted groups toward a data-driven approach that learns how much information to share at each layer.
The DMT CNN Architecture
The core of the proposed framework is the Dynamic Multi-Task CNN (DMT CNN). Unlike monolithic backbones, the DMT CNN assigns a dedicated subnet to each task, ensuring task identity and completeness.
The architecture is defined by several key components:
- Task-Specific Subnets: Each task has its own branch, allowing it to maintain specialized features.
- Task Transfer Connections (TTC): These are learnable parameters ($\alpha$) that link layers across different subnets. Through gradient descent, the model updates these values to determine the strength of information flow. A high $\alpha$ indicates heavy reliance on another task's features, while an $\alpha$ of zero implies independence.
- Dynamic Adjustment: The model "softly" partitions tasks into groups by deciding the degree of sharing at every specific layer, rather than imposing a global sharing constraint.
Dynamic Sharing and Task Relatedness
One of the most compelling features of DMT CNN is its ability to visualize and quantify task relatedness. The authors observed that tasks focusing on similar image regions (e.g., "Goatee" and "Sideburns") naturally develop stronger sharing connections.
As shown in the training curves, the "sharing degree" for various facial attributes converges to stable values over several epochs. This automated "weak grouping" allows a task to draw varying amounts of information from multiple sources, creating a nuanced web of relationships. Furthermore, analysis revealed that semantic sharing becomes more pronounced in deeper layers, suggesting that while low-level features are universal, high-level cross-talk is critical for task-specific performance.
Experimental Results
The DMT CNN was evaluated on the CelebA dataset (40 facial attributes) and CIFAR-10 (reorganized into five binary tasks).
- Accuracy Gains: DMT CNN consistently outperformed Single-Task (ST) and traditional Multi-Task (MT) baselines. For instance, on the highly imbalanced "Bald" attribute, DMT achieved 80.1% accuracy, a massive improvement over the 56.4% achieved by a single-task model.
- Incremental Learning: The architecture supports adding new tasks by creating a new subnet and learning its TTCs to existing, frozen subnets. This process was found to be approximately 4 times faster than training a new model from scratch, as it leverages features already learned by previous tasks.
Conclusion
The Dynamic Multi-Task CNN provides a robust solution to the limitations of manual task grouping in MTL. By learning task relationships through TTCs, it achieves superior accuracy and scalability. While the model currently assumes homogeneous subnet architectures, future work may address the memory overhead of maintaining multiple subnets and explore the integration of heterogeneous structures.
Summary updated