RTS3D: Real-time Stereo 3D Detection from 4D Feature-Consistency Embedding Space for Autonomous Driving

Peixuan Li, Shun Su, Huaici ZhaoAAAI 2021 · DOI

Abstract

RTS3D is the first stereo-based 3D object detection system to achieve true real-time performance (over 24 FPS) while significantly outperforming previous state-of-the-art methods. By replacing traditional, computationally expensive depth-map estimation with a novel 4D Feature-Consistency Embedding (FCE) space, the researchers eliminated the need for pixel-wise depth supervision and instance masks, making the system both faster and easier to train.

Introduction

3D object detection is a cornerstone of autonomous driving. While LiDAR sensors provide high accuracy, they are expensive and bulky. Consequently, researchers have turned to "Pseudo-LiDAR" methods, which use stereo cameras to estimate depth maps and then treat that data like a 3D point cloud. However, Pseudo-LiDAR has two major drawbacks: the high computational cost of generating depth maps and the requirement for massive amounts of pixel-wise ground truth data (depth maps and instance masks) for training.

RTS3D aims to bridge the performance gap between image-based and LiDAR-based methods without the heavy computational and labeling overhead. The core innovation is the 4D Feature-Consistency Embedding (FCE) space. Instead of asking "Is there an object at this 3D coordinate?" (occupancy), the model asks "How consistent are the visual features from the left and right cameras at this 3D coordinate?"

The RTS3D approach leverages a Feature-Consistency Space to achieve high performance with minimal supervision compared to traditional depth-based methods.

Methodology

The RTS3D pipeline consists of four main stages designed to transform stereo images into precise 3D bounding boxes efficiently:

  1. Initial Latent Space Generation: A fast monocular 3D detector (KM3D-Net) identifies potential object locations to create a coarse "latent space" cuboid. This focuses the computation only on relevant areas.
  2. Multi-scale Texture Cues: A Siamese ResNet18 extracts features from the stereo pair at various scales.
  3. FCE Space Building: The model warps left and right features into the 3D latent space. It uses a Semantic-guided Radial Basis Function (RBF) to weight these features, effectively filtering out noise from the background or "non-Lambertian" surfaces (like shiny car paint) without needing explicit instance masks.
  4. 3D Box Prediction: A Structure-aware Attention (StrAA) module captures the object's geometry from a Bird’s Eye View (BEV) perspective. Finally, a PointNet-like regressor predicts the final 3D bounding box and confidence score.
Detailed architecture of the RTS3D pipeline, illustrating the flow from stereo input to the 4D Feature-Consistency Embedding space and final 3D detection.

Experiments and Results

The authors evaluated RTS3D on the KITTI 3D object detection benchmark, the industry standard for autonomous driving.

  • Real-Time Performance: RTS3D achieves 25.4 FPS on a single GPU (2080Ti), making it the first stereo 3D detection system to run in real-time. For comparison, the previous state-of-the-art, Stereo R-CNN, runs at only ~2.4 FPS.
  • Accuracy: On the KITTI benchmark (Moderate setting), RTS3D achieved a 10% improvement in Average Precision (AP) over previous image-only methods.
  • Label Efficiency: The model achieves these results using only 3D bounding box labels for supervision, whereas competing models require depth maps, instance masks, and sometimes even CAD models.

Conclusion

RTS3D demonstrates that feature consistency between stereo views inherently encodes structural information, serving as an efficient proxy for depth. By detecting objects directly in the FCE space, the model avoids the bottleneck of generating standalone depth maps. While the model currently processes frames independently, future work will focus on temporal consistency and smoothing detection results between adjacent video frames to further improve stability.

Summary updated

BibTeX
@inproceedings{li2021rts,
  author = {Peixuan Li and Shun Su and Huaici Zhao},
  title = {{RTS3D: Real-time Stereo 3D Detection from 4D Feature-Consistency Embedding Space for Autonomous Driving}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i3.16288},
}
Figure