AdaMV-MoE: Adaptive Multi-Task Vision Mixture-of-Experts

Tianlong Chen, Xuxi Chen, Xianzhi Du, Abdullah Rashwan, Fan Yang, Huizhong Chen, Zhangyang Wang, Yeqing Li

Introduction

Multi-task learning (MTL) aims to execute multiple computer vision objectives—such as image classification and dense object detection—within a single unified model. While efficient, traditional shared-backbone architectures suffer from "negative competition," where gradient conflicts between disparate tasks hinder convergence. Furthermore, these models typically employ a static capacity, failing to account for the inherent differences in task complexity; for instance, pixel-level segmentation requires significantly more representational power than global image classification.

The AdaMV-MoE Framework

To address these limitations, we present AdaMV-MoE (Adaptive Multi-Task Vision Mixture-of-Experts). This framework replaces standard feed-forward networks in Vision Transformers (ViTs) with sparsely activated Mixture-of-Experts (SMoE) layers governed by two key innovations:

  1. Task-dependent Routing Policies: Unlike global routing mechanisms, AdaMV-MoE assigns a dedicated router network to each task. These routers learn to select the most relevant experts from a shared pool based on the specific task requirements and input content.
  2. Adaptive Expert Selection (AES): AES dynamically determines the optimal number of activated experts ($k$) during training. The process begins with a single expert and monitors validation loss; if performance plateaus, the algorithm activates an additional expert, expanding capacity only as needed to reach a task-specific "sweet spot."
Architecture of the AdaMV-MoE framework, illustrating the transition from standard ViT layers to SMoE layers with task-specific routing.

Experimental Results

The framework was evaluated on the ImageNet-1k (classification) and MS COCO 2017 (detection and segmentation) benchmarks using various backbones, including ViT-Small and UViT-Base.

Performance Gains

AdaMV-MoE consistently outperforms both vanilla MTL baselines and existing state-of-the-art methods like TAPS. On the UViT-Base backbone, the model achieved a 7.39% improvement in ImageNet accuracy and a 1.13% AP increase in COCO object detection compared to standard MTL-ViT models. Notably, it achieves these results while maintaining high inference efficiency by only activating a subset of parameters per task.

Comparison of AdaMV-MoE against baselines and TAPS, showing superior accuracy and detection performance across different model scales.

Key Insights

  • Complexity-Awareness: The model naturally scales its parameters based on difficulty. In ViT-Small configurations, classification utilized ~16.3M parameters, while the more complex detection tasks scaled up to ~19.0M.
  • Reduced Gradient Conflict: Analysis of gradient cosine distances confirms that AdaMV-MoE effectively "disentangles" tasks into different parameter spaces, significantly reducing the tug-of-war between task updates.
  • Content Specialization: Visualizations reveal that experts specialize not only by task but by visual content, with specific experts focusing on background textures versus object boundaries.

Conclusion and Future Work

AdaMV-MoE demonstrates that adaptive capacity is essential for effective multi-task vision systems. By allowing tasks to dynamically claim the "brain power" they require, the model achieves high performance with inference efficiency. While training overhead and expert saturation remain challenges, future research will explore extending this adaptive logic to multi-modal settings, integrating vision with language and audio for more versatile general-purpose AI systems.

Summary updated

Abstract

Sparsely activated Mixture-of-Experts (MoE) is becoming a promising paradigm for multi-task learning (MTL). Instead of compressing multiple tasks’ knowledge into a single model, MoE separates the parameter space and only utilizes the relevant model pieces given task type and its input, which provides stabilized MTL training and ultra-efficient inference. However, current MoE approaches adopt a fixed network capacity (e.g., two experts in usual) for all tasks. It potentially results in the over-fitting of simple tasks or the under-fitting of challenging scenarios, especially when tasks are significantly distinctive in their complexity. In this paper, we propose an adaptive MoE framework for multi-task vision recognition, dubbed AdaMV-MoE. Based on the training dynamics, it automatically determines the number of activated experts for each task, avoiding the laborious manual tuning of optimal model size. To validate our proposal, we benchmark it on ImageNet classification and COCO object detection & instance segmentation which are notoriously difficult to learn in concert, due to their discrepancy. Extensive experiments across a variety of vision transformers demonstrate a superior performance of AdaMV-MoE, compared to MTL with a shared backbone and the recent state-of-the-art (SoTA) MTL MoE approach. Codes are available online: https://github.com/google-research/google-research/tree/master/moe_mtl.
BibTeX
@inproceedings{chen2023adamv,
  author = {Tianlong Chen and Xuxi Chen and Xianzhi Du and Abdullah Rashwan and Fan Yang and Huizhong Chen and Zhangyang Wang and Yeqing Li},
  title = {{AdaMV-MoE: Adaptive Multi-Task Vision Mixture-of-Experts}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.01591},
}
Figure