MultiNet++: Multi-Stream Feature Aggregation and Geometric Loss Strategy for Multi-Task Learning

Sumanth Chennupati, Ganesh Sistu, Senthil Kumar Yogamani, Samir A. RawashdehWorkshops @ CVPR 2019 · DOI

Introduction

In the rapidly evolving field of autonomous driving, a vehicle's perception system must simultaneously perform various computer vision tasks, such as semantic segmentation, depth estimation, and motion detection. While Multi-Task Learning (MTL) offers a path toward computational efficiency by sharing features across these tasks, it faces two significant challenges: the neglect of temporal context in standard single-frame models and the difficulty of balancing losses from tasks with vastly different scales and convergence rates. MultiNet++ addresses these issues through a novel multi-stream architecture and a robust Geometric Loss Strategy (GLS).

Multi-Stream Architecture

Unlike traditional MTL models that process isolated images, MultiNet++ leverages temporal context by utilizing a "Siamese" encoder structure. The network takes two consecutive video frames—the current frame ($t$) and the previous frame ($t-1$)—as input.

The MultiNet++ architecture utilizes a shared Siamese encoder to process consecutive frames, with dedicated aggregation layers feeding task-specific decoders.

As shown in the architecture diagram, both frames pass through a shared ResNet-50 encoder. This design is highly efficient for embedded systems because the features for the previous frame can be cached and reused, resulting in a negligible increase in parameters (approximately 100k for aggregation). The model extracts intermediate feature maps from multiple layers and aggregates them, allowing decoders to perceive environmental changes over time, which is critical for tasks like motion estimation and maintaining boundary consistency.

Geometric Loss Strategy (GLS)

A core contribution of this work is the Geometric Loss Strategy (GLS). Standard MTL approaches typically use a weighted sum of losses ($L_{total} = \sum w_i L_i$), which often requires extensive manual tuning to prevent one task (e.g., depth) from dominating others (e.g., segmentation).

MultiNet++ instead employs the geometric mean of task losses: $$L_{Total} = \sqrt[n]{\prod_{i=1}^{n} L_i}$$

By focusing on the relative change rather than absolute scale, GLS is naturally invariant to the magnitude of individual losses. This mathematical property forces the optimizer to prioritize tasks with higher relative error, ensuring balanced progress across all objectives without complex hyperparameter searches or heavy computational overhead like that found in GradNorm or uncertainty-based weighting.

Experimental Evaluation

The framework was evaluated on three prominent automotive datasets: KITTI, Cityscapes, and SYNTHIA. While KITTI provides real-world data, its sparse sampling makes temporal learning challenging. In contrast, Cityscapes and the synthetic SYNTHIA dataset provide the continuous sequences necessary to fully demonstrate the benefits of multi-stream feature aggregation.

Results and Discussion

MultiNet++ demonstrates superior performance over single-task baselines and other loss-balancing state-of-the-art methods. On the Cityscapes benchmark, the model achieved a ~4% improvement in segmentation accuracy and a ~3% boost in depth estimation.

Quantitative comparison of training loss curves across KITTI and Cityscapes datasets, highlighting the faster convergence and stability of the GLS approach.

As illustrated in the loss curves, the GLS strategy (often represented by the magenta and yellow lines) consistently leads to faster convergence and lower final loss values compared to equal weighting or uncertainty-based methods. Qualitatively, the inclusion of temporal context significantly improved the detection of small, critical objects such as traffic lights and signs, which are frequently blurred in single-frame analysis.

Conclusion

MultiNet++ provides a robust framework for real-time multi-task perception in autonomous vehicles. By combining efficient Siamese feature aggregation with the scale-invariant Geometric Loss Strategy, the authors demonstrate that temporal context can be integrated with minimal overhead to improve both accuracy and training stability. Future work may explore more sophisticated aggregation mechanisms, such as attention modules or Conv-LSTMs, to capture longer-term dependencies in video sequences.

Summary updated

BibTeX
@inproceedings{chennupati2019multinet,
  author = {Sumanth Chennupati and Ganesh Sistu and Senthil Kumar Yogamani and Samir A. Rawashdeh},
  title = {{MultiNet++: Multi-Stream Feature Aggregation and Geometric Loss Strategy for Multi-Task Learning}},
  booktitle = {Workshops @ CVPR},
  year = {2019},
  doi = {10.1109/cvprw.2019.00159},
}
Figure