TL;DR
This paper introduces MonoDVPS, a novel multi-task network for depth-aware video panoptic segmentation that significantly reduces reliance on expensive ground truth data. By combining self-supervised monocular depth estimation with semi-supervised video panoptic segmentation, and introducing innovative panoptic-guided losses and motion masking techniques, MonoDVPS achieves competitive performance with faster inference.
What Problem Does This Paper Address?
Autonomous systems, particularly self-driving cars, require a comprehensive understanding of their surroundings. This involves not only identifying objects (semantic segmentation) and distinguishing individual instances of those objects (instance segmentation) but also understanding their 3D structure and depth. Traditional approaches often rely on expensive multi-modal sensor setups (like LiDAR and cameras) or require vast amounts of meticulously annotated data.
Depth-aware video panoptic segmentation (DVPS) aims to create a holistic 3D representation by combining monocular depth estimation with video panoptic segmentation. However, acquiring precise ground truth for both depth and panoptic segmentation is costly and time-consuming. This paper tackles the challenge of achieving high-performance DVPS while minimizing the need for extensive ground truth annotations, particularly by leveraging unlabeled video sequences.
Methods
MonoDVPS is a multi-task network designed to perform monocular depth estimation, video panoptic segmentation, and instance tracking simultaneously. The core innovation lies in its training paradigm:
- Self-Supervised Monocular Depth Estimation (SSMDE): Instead of relying on ground truth depth, the network learns depth by synthesizing novel views from adjacent frames and minimizing reprojection errors. This is achieved through a geometric projection model and an ego-motion estimation network.
- Semi-Supervised Video Panoptic Segmentation: The network is trained on a combination of human-annotated labels and automatically generated "pseudo-labels" from a pre-trained panoptic segmentation model. This allows it to leverage large amounts of unlabeled video data.
- Panoptic-Guided Depth Losses: To improve depth prediction, the authors introduce three novel loss functions that leverage the strong correlation between panoptic segmentation edges and depth discontinuities:
- Panoptic-guided Smoothness Loss (LPGs): Encourages depth smoothness within panoptic segments.
- Panoptic-guided Edge Discontinuity Loss (LPED): Enforces depth discontinuities at panoptic edges.
- Panoptic-guided Triplet Loss (LPGT): Extends semantic-guided triplet loss to the panoptic domain to better capture depth differences across panoptic contours.
- Moving Objects Masking: A key challenge in SSMDE is the assumption of a static scene. Moving objects introduce photometric errors that corrupt training. The authors propose a novel masking scheme that uses panoptic labels from consecutive frames to identify and exclude pixels corresponding to potentially moving objects from the photometric loss calculation. This is achieved by leveraging optical flow to warp panoptic labels and comparing consistency.
- Multi-Task Learning: The network is trained end-to-end with a carefully balanced combination of loss functions for depth, semantics, instance centers, offsets, and optical flow.
Datasets
The paper uses the following datasets:
- Cityscapes-DVPS: An urban driving dataset that extends Cityscapes with temporally consistent panoptic annotations for every 5th frame in a video snippet. The authors augment the training set by generating pseudo-labels for all frames.
- SemKITTI-DVPS: Based on the odometry split of the KITTI dataset, this dataset provides annotations for every frame, derived from projected 3D point clouds.
Key Insights and Ideas
- Panoptic Edges as Depth Guides: The observation that panoptic contours (edges between instances and stuff) align well with depth discontinuities is a crucial insight. This allows for the development of more effective depth regularization losses than those based solely on semantic edges.
- Leveraging Unlabeled Data is Key: The success of the self-supervised and semi-supervised approaches highlights the immense potential of using large amounts of unlabeled video data to overcome the cost of ground truth annotation.
- Moving Object Handling is Critical for SSMDE: The proposed moving object masking technique is essential for improving the robustness of self-supervised depth estimation in real-world scenarios where scenes are dynamic.
- Multi-task Learning Synergies: Training depth and panoptic segmentation jointly, with panoptic information guiding depth, leads to improved performance for both tasks.
Key Results
- MonoDVPS achieves competitive results on both Cityscapes-DVPS and SemKITTI-DVPS datasets for Depth-aware Video Panoptic Segmentation (DVPQ).
- On Cityscapes-DVPS, MonoDVPS with a ResNet-50 backbone surpasses ViP-DeepLab with the same backbone and sets a new state-of-the-art.
- The proposed panoptic-guided depth losses and moving objects masking significantly improve depth estimation accuracy (e.g., reducing absolute relative error).
- The model offers a good trade-off between accuracy and inference speed, outperforming other methods in terms of speed when using comparable backbones. For instance, MonoDVPS achieves a VPQ of 59.1 with a time of 0.10s on Cityscapes-DVPS, while ViP-DeepLab [22] achieves 63.1 with a time of 0.54s.
Limitations and Future Work
- Supervised vs. Self-Supervised Depth: While the self-supervised depth estimation is significantly improved, fully supervised depth estimation still yields higher accuracy. The authors suggest that further bridging this gap is a direction for future work.
- Heavier Backbones: The performance of MonoDVPS could be further enhanced by using heavier backbones (like WR-41) and employing costly test-time augmentations, similar to ViP-DeepLab.
- Dataset Complexity and Depth Errors: The paper notes that Cityscapes-DVPS is more sensitive to depth error thresholds, suggesting that depth errors are larger on instances than on stuff pixels in more complex scenarios. Future work could focus on improving instance-level depth accuracy.
- Temporal Consistency: While the method leverages video, the core panoptic segmentation is still largely frame-by-frame with instance tracking. More explicit temporal modeling could further enhance video panoptic segmentation.
Summary updated