Sharing Decoders: Network Fission for Multi-task Pixel Prediction

Steven Hickson, Karthik Raveendran, Irfan A. Essa

Abstract

In multi-task learning (MTL) for pixel-wise prediction, the standard approach involves a shared encoder followed by task-specific decoders. This paper introduces the concept of "Network Fission" to explore the optimal point for splitting a network. We demonstrate that sharing the majority of the decoder—a strategy termed Mid-Fission—leads to significant efficiency gains and improved accuracy. Our approach allows a single model to predict semantics, surface normals, and depth simultaneously, with additional tasks requiring as little as 0.017% more FLOPs than a single-task network. On benchmarks like NYUDv2, Mid-Fission achieves state-of-the-art results while reducing computational overhead by nearly 50% compared to standard early-fission architectures.

Introduction

Computer vision applications often require the simultaneous execution of multiple pixel-wise tasks, such as identifying objects (semantic segmentation) and estimating scene geometry (surface normals or depth). The conventional approach, known as Early Fission, uses a shared encoder to extract features and then splits into separate, task-specific decoders. However, these decoders are computationally expensive, making them difficult to deploy on resource-constrained devices like mobile phones. This research investigates the fundamental architectural question: Where is the optimal point to split a network for multi-task learning to maximize both efficiency and performance?

Methods

The authors introduce Network Fission to describe different architectural splitting strategies:

  • Early Fission: Splitting occurs at the encoder bottleneck (the standard approach).
  • Mid-Fission (Proposed): The encoder and the majority of decoder blocks are shared, splitting only at the final intermediate stages.
  • Late Fission: The entire encoder and decoder are shared, splitting only at the very last output layer.
Illustration of different Network Fission strategies: (a) Early, (b) Early-Mid, (c) Mid, and (d) Late Fission.

The study evaluates these strategies using three backbones: a high-performance server model (AdapNet++) and two lightweight mobile models (MobileNetV1-Unet and MobileNetV3-Attention). These are tested across three major benchmarks: SceneNet RGBD (synthetic), NYUDv2 (real-world indoor), and ScanNetv2 (large-scale indoor).

Key Results and Insights

The "Free Lunch" of Mid-Fission

Surprisingly, sharing the decoder does not just save memory; it often improves accuracy. The authors found that Mid-Fission outperforms Early Fission while requiring significantly fewer floating-point operations (FLOPs). Specifically, Mid-Fission requires only 51.9% of the FLOPs of the standard Early-Fission approach. Adding a second or third task to a base model via Mid-Fission adds a negligible 0.017% overhead.

Quantitative comparison of fission strategies and initialization methods on the SceneNet RGBD dataset.

Surface Normals as a Universal Primer

A significant finding is that initializing a multi-task network with weights from a model pre-trained on surface normals consistently yields better results than starting from scratch or ImageNet weights. Surface normals appear to provide a structural "scaffold" that helps the network learn semantics and depth more effectively.

Mobile Deployment

On a Pixel 3XL phone, the Mid-Fission approach reduced latency from 30.1ms to 18.78ms compared to Early Fission, while actually improving semantic accuracy from 81.4% to 84.94%.

Limitations and Future Work

While Mid-Fission works exceptionally well for semantics and surface normals, depth prediction remains more sensitive. Depth often requires more task-specific features (more un-shared layers) to maintain peak accuracy, as it represents absolute distance rather than relative geometry. Future work will explore how Mid-Fission scales to a larger variety of tasks and investigate more sophisticated loss functions to bridge the gap in multi-task depth prediction.

Conclusion

Sharing decoders in multi-task networks is a highly effective strategy for balancing performance and efficiency. Mid-Fission represents a significant step toward deploying sophisticated multi-task vision models on resource-constrained hardware without sacrificing accuracy.

Summary updated

Abstract

We examine the benefits of splitting encoder-decoders for multitask learning and showcase results on three tasks (semantics, surface normals, and depth) while adding very few FLOPS per task. Current hard parameter sharing methods for multi-task pixel-wise labeling use one shared encoder with separate decoders for each task. We generalize this notion and term the splitting of encoder-decoder architectures at different points as fission. Our ablation studies on fission show that sharing most of the decoder layers in multi-task encoder-decoder networks results in improvement while adding far fewer parameters per task. Our proposed method trains faster, uses less memory, results in better accuracy, and uses significantly fewer floating point operations (FLOPS) than conventional multi-task methods, with additional tasks only requiring 0.017% more FLOPS than the single-task network. We show results with a real-time model on a Pixel phone with released source code.
BibTeX
@inproceedings{hickson2022sharing,
  author = {Steven Hickson and Karthik Raveendran and Irfan A. Essa},
  title = {{Sharing Decoders: Network Fission for Multi-task Pixel Prediction}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
  doi = {10.1109/wacv51458.2022.00371},
}
Figure