Abstract
We propose an end-to-end network that takes a single perspective RGB image of a complex road scene as input, to produce occlusion-reasoned layouts in perspective space as well as a parametric bird's-eye-view (BEV) space. In contrast to prior works that require dense supervision such as semantic labels in perspective view, our method only requires human annotations for parametric attributes that are cheaper and less ambiguous to obtain. To solve this challenging task, our design is comprised of modules that incorporate inductive biases to learn occlusion-reasoning, geometric transformation and semantic abstraction, where each module may be supervised by appropriately transforming the parametric annotations. We demonstrate how our design choices and proposed deep supervision help achieve meaningful representations and accurate predictions. We validate our approach on two public datasets, KITTI and NuScenes, to achieve state-of-the-art results with considerably less human supervision.
The Challenge of Road Layout Estimation
For autonomous vehicles to navigate safely, they require a "top-down" understanding of the road—knowing the number of lanes, sidewalk locations, and intersection shapes. Traditionally, this has been approached in two ways: non-parametric methods that produce pixel-level semantic maps, and parametric methods that output structured data (e.g., "3 lanes on the left").
Non-parametric methods require massive amounts of pixel-level human-annotated data, which is slow and expensive. Parametric methods are more interpretable but often struggle with complex scenes or require dense perspective-view supervision. Furthermore, perspective images are riddled with occlusions; parked cars or buildings block the view of essential road features. A robust system must reason about what is behind these obstacles to build a complete map.
Proposed Methodology
The authors propose a three-stage pipeline designed to incorporate inductive biases that guide the model toward understanding geometry and occlusion.
1. Perspective Semantics (PS) Module
Instead of merely identifying visible pixels, this module is trained to predict "occlusion-reasoned" semantics. It learns to infer the road layout even in areas blocked by foreground objects like cars or buildings.
2. Top-view Semantics (TS) Module
This module performs two critical tasks. First, it applies a geometric transformation (assuming a flat road plane) to warp the perspective view into a top-down view. Second, it employs a "Hallucination" network to fill in noisy or missing data caused by the camera's limited field of view or distance.
3. Top-view Parametric Prediction (TPP) Module
Finally, a combination of CNNs and MLPs takes the hallucinated BEV map and regresses the actual road attributes, such as the number of lanes or the distance to side roads.
"Weakly but Deeply Supervised"
The core innovation lies in the training strategy. The authors developed a "renderer" that takes parametric attributes (the "weak" labels) and automatically generates 2D top-down semantic maps. These maps are then projected back into the perspective view. These rendered maps serve as "deep" intermediate supervision for the PS and TS modules, allowing the network to learn complex geometry without requiring human-labeled pixels.
Key Insights and Results
The model was evaluated on the KITTI and NuScenes datasets, utilizing parametric annotations from existing benchmarks to compare "cheap" supervision against "expensive" alternatives.
- Efficiency: Annotating parametric attributes takes approximately 20 seconds per image, compared to over 20 minutes for dense pixel-level labeling—a 60x speedup.
- Hallucination and Context: The TS module’s ability to fill gaps in the BEV map is critical. Similarly, the PS module learns to rely on contextual cues (like road flow) to reason about occluded regions.
- State-of-the-Art Performance: On KITTI, the model achieved an F1 score of 47.3%, outperforming baselines with similar supervision and rivaling models using much more expensive data (e.g., 3D object info).
- Ablation Studies: Removing either the PS or TS modules led to significant performance drops, confirming the necessity of the multi-stage occlusion-reasoning architecture.
Limitations and Future Work
The current model assumes a flat-plane geometry, which may fail in hilly terrain. Additionally, it operates on single images; the authors suggest that incorporating temporal information from video sequences could further improve the hallucination of occluded areas. Future work may also involve training on rich simulated data to further enhance the TPP module.
Summary updated