Introduction
Multi-task learning (MTL) represents a significant frontier in computer vision, enabling a single model to perform diverse tasks such as depth estimation, semantic segmentation, and edge detection. However, deploying these models on resource-constrained edge devices remains a formidable challenge. Traditional MTL architectures often suffer from "negative transfer"—where tasks compete for the same parameters—and high computational overhead, as the entire network must typically be activated regardless of the specific task required.
M3ViT addresses these bottlenecks through a model-accelerator co-design framework. By integrating sparse Mixture-of-Experts (MoE) into a Vision Transformer (ViT) backbone and pairing it with a custom FPGA computation scheme, the framework achieves a massive 88% reduction in inference FLOPs and nearly 10x better energy efficiency than traditional GPU deployments, all while improving task accuracy.
The Challenges of Edge-Based MTL
The development of M3ViT was driven by three primary obstacles:
- Training Conflicts: When a compact model attempts to learn multiple tasks simultaneously, gradient interference often occurs. Task A may require a weight adjustment that contradicts the needs of Task B, leading to suboptimal performance across the board.
- Inference Inefficiency: Most MTL systems are "all-or-nothing." Even if a system only requires a single output (e.g., depth estimation for a robot), the entire backbone designed for multiple tasks must be executed, wasting energy and compute cycles.
- Hardware Bottlenecks: Sparse models like MoE require frequent "expert switching." On standard hardware, constantly loading different expert weights from off-chip memory creates significant latency, often negating the theoretical computational savings of sparsity.
Software Innovation: Task-Dependent MoE
M3ViT replaces the standard dense feed-forward networks (FFN) in a Vision Transformer with MoE layers. The core of this approach lies in its "gating network" or router, which selects a subset of "experts" (small MLP sub-networks) for a given input.
To resolve training conflicts, the researchers introduced two routing variants:
- Multi-gate Routing: Each task is assigned its own dedicated router.
- Task-conditioned Routing: A single router processes task-specific embeddings to select experts.
This "disentanglement" ensures that different tasks utilize different parameters, effectively providing separate "lanes" for each task and preventing gradient interference during the training process.
Hardware Innovation: Computation Reordering
To realize the benefits of MoE on edge hardware like the Xilinx ZCU104 FPGA, M3ViT moves away from traditional token-by-token processing. Standard processing would require reloading expert weights for every token, leading to a "memory wall."
Instead, M3ViT employs Expert-by-Expert Processing. The hardware collects tokens into queues for each expert and processes all tokens assigned to "Expert 1" before switching to "Expert 2." This is supported by a Double-Buffering (ping-pong) scheme: while the FPGA computes results for one expert, it simultaneously pre-fetches the weights for the next expert from memory. This approach effectively hides memory latency, enabling zero-overhead switching.
Experimental Evaluation
The framework was evaluated on the PASCAL-Context and NYUD-v2 datasets, with additional scalability testing on Taskonomy (up to 9 concurrent tasks).
Efficiency and Accuracy
M3ViT demonstrates that sparsity can enhance both performance and efficiency:
- Accuracy: On PASCAL-Context, M3ViT achieved a +2.71% average performance improvement over single-task baselines, outperforming state-of-the-art methods like MTAN and Cross-Stitch.
- FLOPs: The model achieved an 88% reduction in inference FLOPs for single-task execution compared to dense encoder-focused MTL baselines.
Hardware Performance
The co-design approach yielded significant gains on the Xilinx ZCU104:
- Memory: A 2.40× reduction in memory requirements.
- Energy Efficiency: 9.23× higher energy efficiency (latency × power) than comparable FPGA baselines and 10.79× higher efficiency than a high-end NVIDIA Quadro RTX 8000 GPU.
Conclusion
M3ViT proves that the "memory wall" and "task conflict" problems in MTL can be solved through a holistic approach. By treating MoE as both a structural solution for gradient interference and a computational tool for sparsity—and then optimizing the hardware to handle that sparsity—M3ViT sets a new standard for efficient multi-task vision on the edge. Future work will focus on testing these models in unpredictable real-world environments and scaling to even larger task sets.
Summary updated