Abstract
In multi-task learning (MTL) for pixel-wise prediction, the standard approach involves a shared encoder followed by task-specific decoders. This paper introduces the concept of "Network Fission" to explore the optimal point for splitting a network. We demonstrate that sharing the majority of the decoder—a strategy termed Mid-Fission—leads to significant efficiency gains and improved accuracy. Our approach allows a single model to predict semantics, surface normals, and depth simultaneously, with additional tasks requiring as little as 0.017% more FLOPs than a single-task network. On benchmarks like NYUDv2, Mid-Fission achieves state-of-the-art results while reducing computational overhead by nearly 50% compared to standard early-fission architectures.
Introduction
Computer vision applications often require the simultaneous execution of multiple pixel-wise tasks, such as identifying objects (semantic segmentation) and estimating scene geometry (surface normals or depth). The conventional approach, known as Early Fission, uses a shared encoder to extract features and then splits into separate, task-specific decoders. However, these decoders are computationally expensive, making them difficult to deploy on resource-constrained devices like mobile phones. This research investigates the fundamental architectural question: Where is the optimal point to split a network for multi-task learning to maximize both efficiency and performance?
Methods
The authors introduce Network Fission to describe different architectural splitting strategies:
- Early Fission: Splitting occurs at the encoder bottleneck (the standard approach).
- Mid-Fission (Proposed): The encoder and the majority of decoder blocks are shared, splitting only at the final intermediate stages.
- Late Fission: The entire encoder and decoder are shared, splitting only at the very last output layer.
The study evaluates these strategies using three backbones: a high-performance server model (AdapNet++) and two lightweight mobile models (MobileNetV1-Unet and MobileNetV3-Attention). These are tested across three major benchmarks: SceneNet RGBD (synthetic), NYUDv2 (real-world indoor), and ScanNetv2 (large-scale indoor).
Key Results and Insights
The "Free Lunch" of Mid-Fission
Surprisingly, sharing the decoder does not just save memory; it often improves accuracy. The authors found that Mid-Fission outperforms Early Fission while requiring significantly fewer floating-point operations (FLOPs). Specifically, Mid-Fission requires only 51.9% of the FLOPs of the standard Early-Fission approach. Adding a second or third task to a base model via Mid-Fission adds a negligible 0.017% overhead.
Surface Normals as a Universal Primer
A significant finding is that initializing a multi-task network with weights from a model pre-trained on surface normals consistently yields better results than starting from scratch or ImageNet weights. Surface normals appear to provide a structural "scaffold" that helps the network learn semantics and depth more effectively.
Mobile Deployment
On a Pixel 3XL phone, the Mid-Fission approach reduced latency from 30.1ms to 18.78ms compared to Early Fission, while actually improving semantic accuracy from 81.4% to 84.94%.
Limitations and Future Work
While Mid-Fission works exceptionally well for semantics and surface normals, depth prediction remains more sensitive. Depth often requires more task-specific features (more un-shared layers) to maintain peak accuracy, as it represents absolute distance rather than relative geometry. Future work will explore how Mid-Fission scales to a larger variety of tasks and investigate more sophisticated loss functions to bridge the gap in multi-task depth prediction.
Conclusion
Sharing decoders in multi-task networks is a highly effective strategy for balancing performance and efficiency. Mid-Fission represents a significant step toward deploying sophisticated multi-task vision models on resource-constrained hardware without sacrificing accuracy.
Summary updated