Introduction
Multi-task learning (MTL) aims to execute multiple computer vision objectives—such as image classification and dense object detection—within a single unified model. While efficient, traditional shared-backbone architectures suffer from "negative competition," where gradient conflicts between disparate tasks hinder convergence. Furthermore, these models typically employ a static capacity, failing to account for the inherent differences in task complexity; for instance, pixel-level segmentation requires significantly more representational power than global image classification.
The AdaMV-MoE Framework
To address these limitations, we present AdaMV-MoE (Adaptive Multi-Task Vision Mixture-of-Experts). This framework replaces standard feed-forward networks in Vision Transformers (ViTs) with sparsely activated Mixture-of-Experts (SMoE) layers governed by two key innovations:
- Task-dependent Routing Policies: Unlike global routing mechanisms, AdaMV-MoE assigns a dedicated router network to each task. These routers learn to select the most relevant experts from a shared pool based on the specific task requirements and input content.
- Adaptive Expert Selection (AES): AES dynamically determines the optimal number of activated experts ($k$) during training. The process begins with a single expert and monitors validation loss; if performance plateaus, the algorithm activates an additional expert, expanding capacity only as needed to reach a task-specific "sweet spot."
Experimental Results
The framework was evaluated on the ImageNet-1k (classification) and MS COCO 2017 (detection and segmentation) benchmarks using various backbones, including ViT-Small and UViT-Base.
Performance Gains
AdaMV-MoE consistently outperforms both vanilla MTL baselines and existing state-of-the-art methods like TAPS. On the UViT-Base backbone, the model achieved a 7.39% improvement in ImageNet accuracy and a 1.13% AP increase in COCO object detection compared to standard MTL-ViT models. Notably, it achieves these results while maintaining high inference efficiency by only activating a subset of parameters per task.
Key Insights
- Complexity-Awareness: The model naturally scales its parameters based on difficulty. In ViT-Small configurations, classification utilized ~16.3M parameters, while the more complex detection tasks scaled up to ~19.0M.
- Reduced Gradient Conflict: Analysis of gradient cosine distances confirms that AdaMV-MoE effectively "disentangles" tasks into different parameter spaces, significantly reducing the tug-of-war between task updates.
- Content Specialization: Visualizations reveal that experts specialize not only by task but by visual content, with specific experts focusing on background textures versus object boundaries.
Conclusion and Future Work
AdaMV-MoE demonstrates that adaptive capacity is essential for effective multi-task vision systems. By allowing tasks to dynamically claim the "brain power" they require, the model achieves high performance with inference efficiency. While training overhead and expert saturation remain challenges, future research will explore extending this adaptive logic to multi-modal settings, integrating vision with language and audio for more versatile general-purpose AI systems.
Summary updated