Abstract
Vision for autonomous driving is a uniquely challenging problem: the number of tasks required for full scene understanding is large and diverse; the quality requirements on each task are stringent due to the safety-critical nature of the application; and the latency budget is limited, requiring real-time solutions. In this work we address these challenges with QuadroNet, a one-shot network that jointly produces four outputs: 2D detections, instance segmentation, semantic segmentation, and monocular depth estimates in real-time (>60fps) on consumer-grade GPU hardware. On a challenging real-world autonomous driving dataset, we demonstrate an increase of +2.4% mAP for detection, +3.15% mIoU for semantic segmentation, +5.05% [email protected] for instance segmentation and +1.36% in δ < 1.25 for depth prediction over a baseline approach. We also compare our work against other multi-task learning approaches on Cityscapes and demonstrate state-of-the-art results.
Introduction: The Multi-Task Challenge
Autonomous vehicles require a comprehensive understanding of their surroundings, involving both semantic tasks (identifying what things are) and geometric tasks (identifying where they are in 3D space). Traditionally, researchers have faced a "triple threat" of challenges:
- Complexity: The sheer number of diverse tasks required for full scene understanding.
- Stringency: High accuracy requirements due to the safety-critical nature of driving.
- Latency: A very tight computation budget; the vehicle must process data in real-time to ensure safety.
While multi-task learning (MTL) aims to solve these by sharing a single "backbone" network, naively combining tasks often leads to a "competition" for model capacity, which can reduce accuracy. Furthermore, most MTL models treat tasks as independent outputs, failing to utilize the fact that object boundaries and depth discontinuities usually occur at the same spatial locations.
Architecture and Methods
QuadroNet is a one-shot architecture based on a RetinaNet backbone. Its core innovations lie in how it processes features and facilitates cross-task communication to ensure geometric cues inform semantic predictions.
Dense Pixel-Wise Encoder (DPWE)
Instead of standard convolutions, the model uses a series of atrous (dilated) convolutions. This allows the network to maintain a wide field of view to capture long-range context without losing the fine-grained spatial resolution needed for precise segmentation.
Hybrid Depth Estimation
The model quantizes depth into discrete bins using Spacing Increasing Discretization (SID) but adds a continuous depth residual. This allows the network to predict a general depth range and then "fine-tune" the exact distance, significantly improving precision for nearby objects and road textures.
Depth-Aware Instance Segmentation (DAIS)
Standard instance segmentation often struggles to separate overlapping objects of the same class (e.g., two white cars parked bumper-to-bumper). QuadroNet addresses this by using predicted depth logits as a feature for the instance segmentation head. Because sharp jumps in depth typically occur at object edges, the depth data acts as a structural guide to find the exact boundaries of individual instances. This "depth-aware" formulation led to a massive +5.05% mAP jump in instance segmentation accuracy.
Datasets and Evaluation
The study utilizes two primary datasets to validate performance:
- ADD (Autonomous Driving Dataset): A massive internal dataset from Zoox containing over 2 million images (1920x1200 resolution) collected across different cities and weather conditions. It is significantly larger than standard benchmarks like KITTI.
- Cityscapes: A standard industry benchmark used to compare QuadroNet against other state-of-the-art multi-task learning approaches.
Key Results
QuadroNet achieves an inference latency of approximately 21ms (>60 FPS) on an NVIDIA Tesla V100, comfortably exceeding the real-time threshold for autonomous driving.
Compared to a baseline multi-task approach on the ADD dataset, QuadroNet showed:
- +2.4% mAP for 2D detection.
- +3.15% mIoU for semantic segmentation.
- +5.05% [email protected] for instance segmentation.
On Cityscapes, the model achieved 80.73% mIoU, outperforming other popular multi-task models like Kendall et al. and real-time architectures like FC-HarDNet-70. These results demonstrate that forcing the network to reason about depth and instance boundaries in the same output space creates "cross-task consistent" features that improve performance across the board.
Conclusion and Future Work
QuadroNet demonstrates that monocular depth is not just a useful output—it is a powerful feature for sharpening semantic boundaries. While the current model produces 2D bounding boxes and depth maps, future work may include adding an explicit 3D detection head. Additionally, the authors suggest exploring self-supervised depth estimation to reduce the reliance on expensive Lidar labels during training.
Summary updated