Gated3D: Monocular 3D Object Detection From Temporal Illumination Cues

Frank D. Julca-Aguilar, Jason Taylor, Mario Bijelic, Fahim Mannan, Ethan Tseng, Felix HeideICCV 2021 · OpenReview

Abstract

Today's state-of-the-art methods for 3D object detection are based on lidar, stereo, or monocular cameras. Lidar-based methods achieve the best accuracy, but have a large footprint, high cost, and mechanically-limited angular sampling rates, resulting in low spatial resolution at long ranges. Recent approaches based on low-cost monocular or stereo cameras promise to overcome these limitations but struggle in low-light or low-contrast regions as they rely on passive CMOS sensors. In this work, we propose a novel 3D object detection modality that exploits temporal illumination cues from a low-cost monocular gated imager. We propose a novel deep detector architecture, Gated3D, that is tailored to temporal illumination cues from three gated images. Gated images allow us to exploit mature 2D object feature extractors that guide the 3D predictions through a frustum segment estimation. We assess the proposed method on a novel 3D detection dataset that includes gated imagery captured in over 10,000 km of driving data. We validate that our method outperforms state-of-the-art monocular and stereo approaches at long distances. We will release our code and dataset, opening up a new sensor modality as an avenue to replace lidar in autonomous driving.

Introduction: The "Goldilocks" Problem of 3D Sensing

3D object detection is a cornerstone of autonomous vehicle perception. However, existing sensor modalities face significant trade-offs. Lidar provides high accuracy but is prohibitively expensive and suffers from low spatial resolution at long ranges. Conversely, passive monocular and stereo cameras are cost-effective but "depth-blind" in low-light conditions and struggle with long-range depth estimation due to diminishing geometric cues like disparity.

Gated3D introduces a middle ground: gated imaging. This modality uses active laser pulses to capture "slices" of a scene at specific distance ranges, providing active depth cues similar to Lidar but at the low cost and high resolution of a camera.

Methodology: Exploiting Temporal Illumination

The Gated3D architecture is an end-to-end deep learning model designed to extract 3D information from temporal illumination cues.

Gated Imaging and Slicing

Unlike conventional cameras, a gated camera synchronizes a pulsed laser with a high-speed shutter. This allows the sensor to capture specific depth "slices" (e.g., 0-30m, 30-60m, and 60-100m). The relative intensity of an object across these slices serves as a direct, high-resolution cue for its distance.

Gated imaging captures the scene in discrete distance 'slices' (1, 2, and 3), providing active illumination cues.

Architecture Design

The model employs a two-stage approach:

  1. 2D Detection: A ResNet-101 backbone with a Feature Pyramid Network (FPN) identifies 2D regions of interest (ROIs).
  2. 3D Prediction Head: The system crops the raw gated slices corresponding to 2D ROIs. A two-stream network then disentangles semantic features from depth cues.
  3. Frustum Segment Estimation: To stabilize 3D coordinate prediction, the model estimates a "frustum segment"—a 3D volume where the object is likely located based on its 2D height and camera focal length. The network learns to predict the precise offset within this segment.

The Gated3D Dataset

To validate this approach, the authors introduced a massive benchmark dataset featuring over 10,000 km of driving data from Northern Europe. The dataset includes 1.4 million frames, with 12,997 manually annotated images containing over 100,000 labeled objects. Crucially, it includes synchronized Lidar and stereo RGB data, enabling direct comparisons across modalities in challenging conditions like night, snow, and fog.

Results and Discussion

Gated3D demonstrates significant advantages over traditional passive sensors and even Lidar in specific scenarios.

Full comparison of sensor modalities: RGB (left), gated slices (middle-left), Gated3D detections (middle-right), and Lidar reference (right).

Performance at Range

In the 50-80m range, Gated3D significantly outperforms state-of-the-art monocular (M3D-RPN) and stereo (Stereo-RCNN) methods. For nighttime car detection, Gated3D achieved a 3D Average Precision (AP) of 12.97, compared to just 2.73 for the monocular baseline.

Beating Lidar for Small Objects

While Lidar is generally the gold standard, its point cloud becomes extremely sparse at long distances. Gated3D's high-resolution slices allow it to outperform Lidar-based methods (like PointPillars) for small objects like pedestrians at distance (14.14 AP vs. 0.00 AP for Lidar).

Limitations and Future Directions

The current model runs at approximately 10 FPS on an Nvidia RTX 2080. While promising, further optimization is required for high-speed autonomous applications. Additionally, the frustum segment method assumes standard object heights, which may lead to errors for atypical objects. Future work aims to explore "Gated Stereo," combining active illumination with stereo geometric cues.

Conclusion

Gated3D proves that temporal illumination cues from gated imaging offer a viable, high-resolution, and low-cost alternative for 3D object detection, particularly in environments where traditional sensors fail.

Summary updated

Abstract

Today’s state-of-the-art methods for 3D object detection are based on lidar, stereo, or monocular cameras. Lidar-based methods achieve the best accuracy, but have a large footprint, high cost, and mechanically-limited angular sampling rates, resulting in low spatial resolution at long ranges. Recent approaches using low-cost monocular or stereo cameras promise to overcome these limitations but struggle in low-light or low-contrast regions as they rely on passive CMOS sensors. We propose a novel 3D object detection modality that exploits temporal illumination cues from a low-cost monocular gated imager. We introduce a novel deep detection architecture, Gated3D, that is tailored to temporal illumination cues in gated images. This modality allows us to exploit mature 2D object feature extractors that guide the 3D predictions through a frustum segment estimation. We assess the proposed method experimentally on a 3D detection dataset that includes gated images captured over 10,000 km of driving data. We validate that our method outperforms state-of-the-art monocular and stereo methods, opening up a new sensor modality as an avenue to replace lidar in autonomous driving. https://light.princeton.edu/gated3d
BibTeX
@inproceedings{julcaaguilar2021gated,
  author = {Frank D. Julca-Aguilar and Jason Taylor and Mario Bijelic and Fahim Mannan and Ethan Tseng and Felix Heide},
  title = {{Gated3D: Monocular 3D Object Detection From Temporal Illumination Cues}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=6G8JNWfQj-j},
}
Figure