Learning 2D to 3D Lifting for Object Detection in 3D for Autonomous Vehicles

Siddharth Srivastava, Frédéric Jurie, Gaurav Sharma

TL;DR

This paper introduces a novel approach to 3D object detection for autonomous vehicles by learning to "lift" 2D monocular images into 3D representations. By training generative models (GANs) to create Bird's Eye View (BEV) images from 2D inputs, they enable existing 3D object detectors to perform comparably to, and sometimes better than, methods relying on expensive 3D sensors like LiDAR, while also offering a robust backup system.

What Problem Does This Paper Address?

Autonomous driving systems heavily rely on accurate 3D object detection to perceive their surroundings and navigate safely. Traditionally, this has been achieved through two main avenues: either inferring 3D information from 2D images or directly using 3D sensor data (like LiDAR). While 3D sensors offer rich spatial information, they are expensive, bulky, and can be prone to failure. Methods relying solely on monocular 2D images have historically lagged behind in performance due to the inherent difficulty of inferring depth and spatial relationships from a single viewpoint. This paper aims to bridge this performance gap by leveraging the strengths of both approaches: using readily available 2D images and sophisticated generative models to create 3D-like representations that can be processed by existing 3D detection frameworks, thereby reducing reliance on costly 3D hardware.

Methods

The core of this work lies in generating 3D representations from 2D images. Specifically, the authors propose using Generative Adversarial Networks (GANs), termed "BirdGAN," to translate monocular RGB images into Bird's Eye View (BEV) images. These generated BEV images are designed to be compatible with existing 3D object detection architectures like BirdNet and MV3D, which typically process BEV representations derived from LiDAR data.

The process involves:

  1. BEV Image Generation: A GAN architecture, built upon a VGG-16 encoder and a DCGAN decoder, is trained to generate multi-channel BEV images (e.g., height, density, intensity) from input RGB images.
  2. Training Data Curation: To improve GAN training, the authors experiment with two strategies: using all objects in the scene or focusing on "well-defined" (closer, less occluded) objects. The latter, termed "clipping," aims to reduce noise and improve the quality of generated data.
  3. Integration with 3D Detectors: The generated BEV images are then fed into pre-trained 3D object detection networks (BirdNet, MV3D). For BirdNet, which requires ground plane estimation, a separate network is used to reconstruct a 3D model from the generated BEV to estimate the ground plane. For MV3D, multiple GANs are used to generate the necessary BEV channels, and an auxiliary network generates a front-view image.
  4. Late Fusion: To further boost performance, the authors explore a late fusion strategy where the outputs of a network trained on real 3D data are combined with the outputs of a network trained on generated 3D data.

Datasets

The primary dataset used for training and evaluation is the KITTI Object Detection Benchmark. The authors split the KITTI training set into training and validation sets, following common practice. They do not introduce a new benchmark dataset.

Key Insights and Ideas

  • Feasibility of 2D-to-3D Lifting: The paper convincingly demonstrates that it's not only possible but also highly effective to learn a mapping from 2D images to 3D representations that can be used by existing 3D detectors. This challenges the notion that explicit 3D sensor data is always superior.
  • "Clipping" for Better GAN Training: Focusing GAN training on "well-defined" objects (closer to the camera, less occluded) significantly improves the quality of generated BEV images and, consequently, the performance of the downstream 3D detectors. This highlights the importance of data quality over sheer quantity in generative tasks.
  • Generative Models as Sensor Backups: The proposed method offers a compelling solution for scenarios where 3D sensors might fail or are unavailable, acting as a reliable backup system. This has significant implications for safety and cost-effectiveness in autonomous systems.
  • Generalizability of Generated Data: The generated BEV images are shown to be effective with different 3D detection architectures (BirdNet and MV3D), suggesting the approach is generalizable and can be integrated into various pipelines.
  • Late Fusion Boosts Performance: Combining the outputs of networks trained on real and generated data through late fusion provides a performance improvement over using either individually, demonstrating the synergistic potential of combining real and synthetic data.

Key Results

  • Outperforming Monocular Methods: The proposed 2D-to-3D lifted methods significantly outperform state-of-the-art 3D object detection methods that rely solely on monocular images. For example, with BirdNet and IoU=0.5, they achieve a 27.9% improvement on easy images and 19.54% on hard examples compared to Mono3D.
  • Competitive with 3D Sensor Methods: The method achieves performance comparable to, and in some cases better than, methods that use actual 3D sensor data (LiDAR). For instance, with MV3D and clipped data, they achieve AP3D scores that are competitive with or exceed those of methods using real 3D data.
  • "Clipping" Improves Performance: Training with "clipped" data (focusing on closer objects) consistently leads to better performance, with improvements of 10-25% in APloc and significant gains in AP3D.
  • Late Fusion Gains: Deep fusion of networks trained on real and generated data leads to a 2-5% improvement in performance for both 3D localization and detection on easy and moderate examples.

Limitations and Future Work

  • Hard Examples and Occlusions: The paper notes that performance can drop on hard examples with high IoU thresholds (e.g., IoU=0.7) when objects are heavily occluded or far away. This is attributed to the GAN's difficulty in generating accurate BEVs for such challenging scenarios.
  • Camera Parameter Sensitivity: The authors mention that future work could explore more challenging settings with unknown camera parameters, suggesting that current methods might be sensitive to precise camera calibration.
  • Potential for Redundancy: While not a limitation, the authors suggest that this method could potentially make expensive 3D sensors redundant in certain low-risk autonomous navigation scenarios like warehouse automation.

Summary updated

Abstract

We address the problem of 3D object detection from 2D monocular images in autonomous driving scenarios. We propose to lift the 2D images to 3D representations using learned neural networks and leverage existing networks working directly on 3D data to perform 3D object detection and localization. We show that, with carefully designed training mechanism and automatically selected minimally noisy data, such a method is not only feasible, but gives higher results than many methods working on actual 3D inputs acquired from physical sensors. On the challenging KITTI benchmark, we show that our 2D to 3D lifted method outperforms many recent competitive 3D networks while significantly outperforming previous state-of-the-art for 3D detection from monocular images. We also show that a late fusion of the output of the network trained on generated 3D images, with that trained on real 3D images, improves performance. We find the results very interesting and argue that such a method could serve as a highly reliable backup in case of malfunction of expensive 3D sensors, if not potentially making them redundant, at least in the case of low human injury risk autonomous navigation scenarios like warehouse automation.
BibTeX
@inproceedings{srivastava2019learning,
  author = {Siddharth Srivastava and Frédéric Jurie and Gaurav Sharma},
  title = {{Learning 2D to 3D Lifting for Object Detection in 3D for Autonomous Vehicles}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=p0t8pUlQsT},
}

Loading related content...

Figure