Mono Lay out: Amodal scene layout from a single image

Kaustubh Mani, Swapnil Daga, Shubhika Garg, Narasimhan Sai Shankar, J. Krishna Murthy, K. Madhava Krishna

TL;DR

This paper introduces MonoLayout, a novel deep neural network that estimates the complete 3D layout of a driving scene (roads, sidewalks, and vehicles) from a single 2D image, even for occluded areas. It achieves state-of-the-art performance and real-time speeds, making it a significant step towards robust monocular perception for autonomous driving.

What Problem Does This Paper Address?

Autonomous driving systems rely heavily on understanding the surrounding environment. While powerful sensors like LiDAR are common, there's a strong desire to achieve robust perception using only monocular cameras, which are cheaper and more ubiquitous. A key challenge is that a single image inherently loses 3D information due to projection, and many objects are partially or fully occluded. This paper tackles the problem of "amodal scene layout estimation" – predicting the full 3D layout of static (road, sidewalk) and dynamic (vehicles) elements in a bird's-eye view, going beyond what's directly visible in the image to "hallucinate" occluded regions.

Methods

MonoLayout is a deep neural network designed for this amodal scene layout estimation. Its architecture consists of:

  • Context Encoder: A ResNet-18 based feature extractor that processes the input image to capture multi-scale representations, forming a "shared context."
  • Amodal Layout Decoders: Two parallel decoders (one for static elements like road/sidewalk, one for dynamic elements like vehicles) that take the shared context and predict an occupancy grid in bird's-eye view.
  • Adversarial Feature Learning: Two discriminators are employed to regularize the predicted layouts. These discriminators are trained to distinguish between real (ground truth) and generated (predicted) scene layouts, encouraging the network to produce more realistic and plausible completions, especially for occluded areas.
  • Temporal Sensor Fusion for Training: To generate robust training labels, especially for occluded regions where direct 3D information might be missing, the authors leverage temporal sensor fusion. This involves aggregating sensor data (like monocular depth estimates or lidar scans) over a window of frames to create a denser, noise-free pointcloud and then projecting it into an occupancy grid.

Datasets

The paper evaluates MonoLayout on several standard datasets:

  • KITTI Raw: Used for evaluating static layout estimation.
  • KITTI Odometry: Used for evaluating static layout estimation with temporal sensor fusion.
  • KITTI Object: Used for evaluating dynamic (vehicle) layout estimation.
  • KITTI Tracking: Used for evaluating the full amodal scene layout estimation (static + dynamic).
  • Argoverse: A high-resolution dataset with semantic occupancy grids, also used for full amodal scene layout estimation.

The authors also mention generating training labels using monocular depth estimation and temporal sensor fusion, which can be applied to datasets lacking direct 3D ground truth.

Key Insights and Ideas

  • Amodal Reasoning is Crucial: Humans can infer occluded object parts and scene context effortlessly. Modern neural networks, while excelling at recognition, often struggle with this "beyond-the-image" reasoning. MonoLayout directly addresses this gap.
  • Adversarial Learning for Hallucination: Adversarial training is key to "hallucinating" plausible completions for occluded regions, leading to sharper and more realistic scene layouts compared to purely supervised methods.
  • Shared Context for Joint Estimation: By using a shared context encoder for both static and dynamic scene elements, MonoLayout can leverage correlations between them (e.g., vehicles drive on roads), leading to better overall performance than separate models.
  • Temporal Fusion for Robust Labels: Generating training data by fusing information over time is a powerful technique, especially when dealing with noisy or incomplete sensor data, and it significantly boosts performance.

Key Results

  • State-of-the-Art Performance: MonoLayout significantly outperforms existing approaches on amodal scene layout estimation, achieving margins of over 20% on metrics like mIoU.
  • Real-time Performance: The model achieves a real-time inference rate of approximately 32 Hz on an NVIDIA GeForce GTX 1080Ti GPU, making it practical for on-road applications.
  • Improved Object Detection: MonoLayout also achieves state-of-the-art performance in bird's-eye view object detection without requiring post-processing steps like non-maximum suppression.
  • Qualitative Improvements: Figures in the paper demonstrate MonoLayout's ability to produce crisp object boundaries and accurately complete occluded road and vehicle regions, even in challenging conditions like low light and shadows.

Limitations and Future Work

  • Sidewalk Hallucination: The paper notes that while road hallucination is strong, sidewalk completion can sometimes have artifacts, possibly due to the lack of readily available ground-truth data for sidewalks in training.
  • Failure Cases: High dynamic range, shadows, and sharp turns can sometimes lead to incorrect layout estimates. Multiple nearby vehicles can also be merged into a single estimate in some challenging scenarios.
  • Future Work: The authors suggest exploring generalization to unseen scenarios and incorporating temporal information more deeply to improve performance.

Summary updated

Abstract

In this paper, we address the novel, highly challenging problem of estimating the layout of a complex urban driving scenario. Given a single color image captured from a driving platform, we aim to predict the bird's eye view layout of the road and other traffic participants. The estimated layout should reason beyond what is visible in the image, and compensate for the loss of 3D information due to projection. We dub this problem amodal scene layout estimation, which involves hallucinating scene layout for even parts of the world that are occluded in the image. To this end, we present MonoLayout, a deep neural network for realtime amodal scene layout estimation from a single image. We represent scene layout as a multi-channel semantic occupancy grid, and leverage adversarial feature learning to hallucinate " plausible completions for occluded image parts. We extend several state-of-the-art approaches for road-layout estimation and vehicle occupancy estimation in bird's eye view to the amodal setup and thoroughly evaluate against them. By leveraging temporal sensor fusion to generate training labels, we significantly outperform current art over a number of datasets.
BibTeX
@inproceedings{mani2020mono,
  author = {Kaustubh Mani and Swapnil Daga and Shubhika Garg and Narasimhan Sai Shankar and J. Krishna Murthy and K. Madhava Krishna},
  title = {{Mono Lay out: Amodal scene layout from a single image}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
}
Figure