Abstract
Given an image or a video captured from a monocular camera, amodal layout estimation is the task of predicting semantics and occupancy in bird's eye view. The term amodal implies we also reason about entities in the scene that are occluded or truncated in image space. While several recent efforts have tackled this problem, there is a lack of standardization in task specification, datasets, and evaluation protocols. We address these gaps with AutoLay, a dataset and benchmark for amodal layout estimation from monocular images. AutoLay encompasses driving imagery from two popular datasets: KITTI and Argoverse. In addition to fine-grained attributes such as lanes, sidewalks, and vehicles, we also provide semantically annotated 3D point clouds. We implement several baselines and bleeding edge approaches, and release our data and code.
Introduction
In the field of autonomous driving, "seeing" involves more than just identifying objects in a camera's direct line of sight. To navigate safely, a vehicle must understand the entire scene in a top-down, Bird’s-Eye View (BEV), even when significant portions are occluded by other vehicles or infrastructure. This capability, known as amodal layout estimation, requires the system to reason about and "hallucinate" the geometry of hidden areas.
Problem Statement
Existing research in amodal perception has been hindered by three primary issues:
- Lack of Granularity: Most models only predict coarse categories (e.g., "road" vs. "non-road"), failing to provide the lane-level detail necessary for complex driving maneuvers.
- Temporal Inconsistency: Processing images frame-by-frame often leads to "flickering" predictions where the road layout changes erratically over time.
- Evaluation Gaps: There has been a lack of standardized, large-scale benchmarks providing ground-truth labels for both static layouts and dynamic objects in a unified BEV space.
Methodology: VideoLayout
To address these challenges, we introduce VideoLayout, a neural network architecture that leverages temporal sequences to produce stable and accurate environment maps.
Architecture
VideoLayout consists of a feature encoder that extracts multi-scale visual features from a series of RGB images. These features are processed by a Stacked ConvLSTM, which acts as the system's "memory," aggregating information over time to maintain a consistent mental map of the environment. The model uses separate decoders for static elements (lanes, sidewalks) and dynamic elements (vehicles). A refinement network, utilizing a GAN-based patch discriminator, ensures that the generated layouts adhere to realistic road geometries.
Consistency Losses
To mitigate temporal flickering, we implement short-range and long-range consistency losses. These losses penalize the model if its predictions for specific spatial patches change drastically between consecutive frames or across longer sequences, ensuring physical continuity.
The AutoLay Dataset
A core contribution of this work is the AutoLay Dataset, built upon the KITTI and Argoverse datasets. It includes over 16,000 images covering 12 kilometers of driving, with per-frame annotations in perspective view, 3D point clouds, and orthographic BEV. The dataset features fine-grained categories, including ego-lanes, adjacent lanes, sidewalks, and crosswalks.
Results
VideoLayout establishes a new state-of-the-art on the AutoLay benchmark. On the KITTI dataset, it achieved a 68.73 mIoU for road layout, significantly outperforming previous single-image methods. The improvement in lane detection was particularly notable, with Average Precision (AP) increasing from 16.09 to 38.56.
Despite its temporal complexity, the model maintains high efficiency, running at over 40 FPS on an NVIDIA 1080Ti. Qualitative results demonstrate that VideoLayout produces much smoother and more physically plausible hallucinations of occluded areas compared to non-temporal baselines.
Conclusion
AutoLay provides a much-needed benchmark for fine-grained amodal layout estimation. Our proposed VideoLayout model demonstrates that temporal context is key to overcoming occlusion and ensuring consistency in BEV mapping. Future work will focus on improving robustness under extreme lighting conditions and exploring more complex transformer-based temporal architectures.
Summary updated