Abstract
Solving multiple visual tasks using individual models can be resource-intensive, while multi-task learning can conserve resources by sharing knowledge across different tasks. Despite the benefits of multi-task learning, such techniques can struggle with balancing the loss for each task, leading to potential performance degradation. We present a novel computation- and parameter-sharing framework that balances efficiency and accuracy to perform multiple visual tasks utilizing individually-trained single-task transformers. Our method is motivated by transfer learning schemes to reduce computational and parameter storage costs while maintaining the desired performance. Our approach involves splitting the tasks into a base task and the other sub-tasks, and sharing a significant portion of activations and parameters/weights between the base and sub-tasks to decrease inter-task redundancies and enhance knowledge sharing. The evaluation conducted on NYUD-v2 and PASCAL-context datasets shows that our method is superior to the state-of-the-art transformer-based multi-task learning techniques with higher accuracy and reduced computational resources. Moreover, our method is extended to video stream inputs, further reducing computational costs by efficiently sharing information across the temporal domain as well as the task domain.
Introduction
Performing multiple dense visual tasks—such as semantic segmentation, depth estimation, and edge detection—simultaneously is a cornerstone of modern autonomous systems and augmented reality. Traditionally, this is handled by running separate, specialized models for each task, which is prohibitively expensive in terms of both computation (FLOPs) and memory. While Multi-Task Learning (MTL) aims to solve this by sharing a backbone, it often suffers from "negative transfer," where optimizing for one task degrades performance on another.
This paper introduces EfficientMTL, a framework that achieves the efficiency of shared representations without the optimization difficulties of standard MTL. By leveraging the inherent redundancies between tasks and across time in video streams, the proposed method significantly reduces resource requirements while maintaining or exceeding single-task performance.
Methodology: Computation Sharing via Deltas
The core innovation of EfficientMTL is the use of "delta" representations for both weights and activations. The architecture designates one task as the Base Task (typically semantic segmentation) and treats others as Sub-tasks.
Delta Weights and Activations
Instead of a full set of parameters, each sub-task utilizes the base task's weights $W^{b_1}$ plus a sparse delta matrix $\delta W^{b_k}$. During inference, sub-tasks "borrow" the pre-computed activations $X^{b_1}$ from the base task, computing only the sparse difference $\delta X$ required for their specific output. This sparsity is explicitly enforced during training using $\ell_0$ and $\ell_1$ regularization, ensuring that the majority of the computation is shared.
Temporal and Layer-wise Sharing
For video processing, the framework extends this logic to the temporal domain. If the difference between consecutive frames is minimal, the model reuses activations from the previous time step. A key finding is that task-specific redundancy is higher in early layers, whereas temporal redundancy is more pronounced in deeper layers. EfficientMTL implements a switching boundary to optimize which sharing mechanism is used at each depth of the Transformer.
Experimental Results
The framework was rigorously evaluated on the NYUD-v2 and PASCAL-Context datasets. EfficientMTL was compared against InvPT, a state-of-the-art Transformer-based MTL model.
Efficiency and Accuracy
On the PASCAL-Context dataset, EfficientMTL reduced FLOPs by 49.4% and parameters by 74.0% compared to a single-task baseline. Crucially, it achieved these savings while outperforming existing MTL methods in accuracy. The marginal cost of adding a new sub-task is remarkably low, requiring only ~37% of the FLOPs of a full standalone model.
Sparsity Analysis
The success of the method stems from the high degree of sparsity achieved in the delta components. As shown in the evaluation across various tasks, both weights and activations exhibit significant redundancy relative to the base task, often exceeding 90% sparsity for weights.
Conclusion and Future Work
EfficientMTL demonstrates that task correlation is sparse and can be exploited for massive efficiency gains. By focusing on the unique "deltas" of each task and frame, the framework bypasses the optimization hurdles of traditional MTL. While the theoretical FLOP reductions are substantial, fully realizing these gains in real-time latency will require specialized hardware or software libraries optimized for sparse matrix operations. Future work will focus on developing sparsity-aware Transformer accelerators to bridge this gap between theoretical efficiency and hardware deployment.
Code available at: https://github.com/sarashoouri/EfficientMTL
Summary updated