MonoLayout: Amodal scene layout from a single image

Kaustubh Mani, Swapnil Daga, Shubhika Garg, Narasimhan Sai Shankar, Krishna Murthy Jatavallabhula, K. Madhava Krishna

TL;DR

This paper introduces MonoLayout, a novel deep learning architecture that estimates the complete "amodal" scene layout (road, sidewalks, and vehicles) from a single monocular image, even for occluded areas. This is crucial for autonomous driving systems that need a comprehensive understanding of their surroundings, not just what's directly visible.

What Problem Does This Paper Address?

Autonomous driving systems rely heavily on understanding the 3D scene around them. While powerful sensors like LiDAR can provide rich 3D information, there's a strong desire to achieve robust perception using only monocular cameras, which are cheaper and more ubiquitous. A key challenge is that a single image inherently loses depth information due to projection, and objects can be occluded by others. Humans, however, possess an "amodal" perception, meaning they can infer the complete shape and context of objects even when parts are hidden. This paper tackles the problem of "amodal scene layout estimation" – predicting the full bird's-eye view layout of static elements (road, sidewalks) and dynamic elements (vehicles), including their occluded portions, from a single image.

Methods

MonoLayout is a deep neural network designed for this task. Its architecture consists of:

  • Context Encoder: A ResNet-18 backbone extracts multi-scale features from the input image, creating a "shared context" that captures essential scene information.
  • Static and Dynamic Layout Decoders: Two parallel decoders, with identical architectures, take the shared context and predict the amodal layout for static elements (road/sidewalk) and dynamic elements (vehicles) respectively. These decoders use deconvolutional layers to upsample the feature map into a bird's-eye view occupancy grid.
  • Adversarial Feature Learning: To "hallucinate" plausible completions for occluded regions, MonoLayout employs two discriminators. These discriminators are trained to distinguish between real scene layouts (derived from sources like OpenStreetMap for roads) and generated layouts, pushing the decoders to produce more realistic and complete predictions. This adversarial approach is key to overcoming the limitations of purely supervised learning on incomplete ground truth.
  • Temporal Sensor Fusion for Training: To generate robust training labels for amodal estimation, the authors leverage temporal sensor fusion. This involves aggregating sensor data (like monocular depth estimates or LiDAR scans) over multiple frames to create a denser, noise-free pointcloud and then projecting it into an occupancy grid. This allows for training even when direct 3D information is not available for every frame.

Datasets

The paper evaluates MonoLayout on several established datasets:

  • KITTI Raw: Used for evaluating static layout estimation.
  • KITTI Odometry: Used for evaluating static layout estimation with temporal sensor fusion.
  • KITTI Object: Used for evaluating dynamic (vehicle) layout estimation.
  • KITTI Tracking: Used for evaluating the full amodal scene layout estimation (both static and dynamic).
  • Argoverse: A high-resolution dataset that provides semantic occupancy grids and vehicle detections, facilitating amodal scene layout estimation.

The authors also generated their own training labels using temporal sensor fusion, which is a novel contribution for creating amodal ground truth.

Key Insights and Ideas

  • Amodal Perception is Crucial: The paper strongly argues that for robust autonomous driving, systems need to go beyond visible information and infer occluded parts of the scene, mimicking human amodal perception.
  • Adversarial Learning for Hallucination: Adversarial training is shown to be highly effective in enabling the network to "hallucinate" plausible completions for occluded regions, significantly improving layout estimation accuracy.
  • Shared Context for Joint Estimation: Using a shared context encoder for both static and dynamic layout decoders allows the model to learn interdependencies between road layout and vehicle placement, leading to better overall performance.
  • Temporal Fusion for Robust Labels: Generating training labels through temporal sensor fusion is a practical and effective way to create amodal ground truth, especially when direct 3D sensing is not consistently available.

Key Results

  • State-of-the-Art Performance: MonoLayout significantly outperforms existing methods on amodal scene layout estimation across multiple datasets (KITTI and Argoverse), often by a margin of over 20%.
  • Real-time Performance: The model achieves real-time inference rates (around 32 FPS on an NVIDIA GeForce GTX 1080Ti GPU), making it suitable for on-road applications. This is a substantial speedup compared to many prior methods.
  • Improved Object Detection: MonoLayout also achieves state-of-the-art object detection performance in bird's-eye view without requiring post-processing steps like non-maximum suppression.
  • Qualitative Improvements: Visualizations show MonoLayout producing sharper road and vehicle boundaries and more accurate completions of occluded areas compared to baselines.

Limitations and Future Work

  • Sidewalk Artifacts: The paper notes that due to the lack of readily available ground-truth data for sidewalks, MonoLayout can sometimes produce artifacts in sidewalk predictions.
  • Adverse Conditions: High dynamic range, shadows, and sharp turns can sometimes affect sidewalk estimation accuracy.
  • Vehicle Merging: In cases with multiple nearby vehicles, the model can sometimes merge them into a single estimate, though this is improved when the ego-vehicle is close to the other cars.
  • Future Work: The authors suggest exploring generalization to unseen scenarios and incorporating temporal information more deeply to improve performance. They also highlight the potential for MonoLayout as a foundation for downstream tasks like multi-object tracking and trajectory forecasting.

Summary updated

Abstract

In this paper, we address the novel, highly challenging problem of estimating the layout of a complex urban driving scenario. Given a single color image captured from a driving platform, we aim to predict the bird's-eye view layout of the road and other traffic participants. The estimated layout should reason beyond what is visible in the image, and compensate for the loss of 3D information due to projection. We dub this problem amodal scene layout estimation, which involves "hallucinating" scene layout for even parts of the world that are occluded in the image. To this end, we present MonoLayout, a deep neural network for real-time amodal scene layout estimation from a single image. We represent scene layout as a multi-channel semantic occupancy grid, and leverage adversarial feature learning to hallucinate plausible completions for occluded image parts. Due to the lack of fair baseline methods, we extend several state-of-the-art approaches for road-layout estimation and vehicle occupancy estimation in bird's-eye view to the amodal setup for rigorous evaluation. By leveraging temporal sensor fusion to generate training labels, we significantly outperform current art over a number of datasets. On the KITTI and Argoverse datasets, we outperform all baselines by a significant margin. We also make all our annotations, and code publicly available. A video abstract of this paper is available https://www.youtube.com/watch?v=HcroGyo6yRQ .
BibTeX
@inproceedings{mani2020monolayout,
  author = {Kaustubh Mani and Swapnil Daga and Shubhika Garg and Narasimhan Sai Shankar and Krishna Murthy Jatavallabhula and K. Madhava Krishna},
  title = {{MonoLayout: Amodal scene layout from a single image}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
}
Figure