Abstract
RTS3D is the first stereo-based 3D object detection system to achieve true real-time performance (over 24 FPS) while significantly outperforming previous state-of-the-art methods. By replacing traditional, computationally expensive depth-map estimation with a novel 4D Feature-Consistency Embedding (FCE) space, the researchers eliminated the need for pixel-wise depth supervision and instance masks, making the system both faster and easier to train.
Introduction
3D object detection is a cornerstone of autonomous driving. While LiDAR sensors provide high accuracy, they are expensive and bulky. Consequently, researchers have turned to "Pseudo-LiDAR" methods, which use stereo cameras to estimate depth maps and then treat that data like a 3D point cloud. However, Pseudo-LiDAR has two major drawbacks: the high computational cost of generating depth maps and the requirement for massive amounts of pixel-wise ground truth data (depth maps and instance masks) for training.
RTS3D aims to bridge the performance gap between image-based and LiDAR-based methods without the heavy computational and labeling overhead. The core innovation is the 4D Feature-Consistency Embedding (FCE) space. Instead of asking "Is there an object at this 3D coordinate?" (occupancy), the model asks "How consistent are the visual features from the left and right cameras at this 3D coordinate?"
Methodology
The RTS3D pipeline consists of four main stages designed to transform stereo images into precise 3D bounding boxes efficiently:
- Initial Latent Space Generation: A fast monocular 3D detector (KM3D-Net) identifies potential object locations to create a coarse "latent space" cuboid. This focuses the computation only on relevant areas.
- Multi-scale Texture Cues: A Siamese ResNet18 extracts features from the stereo pair at various scales.
- FCE Space Building: The model warps left and right features into the 3D latent space. It uses a Semantic-guided Radial Basis Function (RBF) to weight these features, effectively filtering out noise from the background or "non-Lambertian" surfaces (like shiny car paint) without needing explicit instance masks.
- 3D Box Prediction: A Structure-aware Attention (StrAA) module captures the object's geometry from a Bird’s Eye View (BEV) perspective. Finally, a PointNet-like regressor predicts the final 3D bounding box and confidence score.
Experiments and Results
The authors evaluated RTS3D on the KITTI 3D object detection benchmark, the industry standard for autonomous driving.
- Real-Time Performance: RTS3D achieves 25.4 FPS on a single GPU (2080Ti), making it the first stereo 3D detection system to run in real-time. For comparison, the previous state-of-the-art, Stereo R-CNN, runs at only ~2.4 FPS.
- Accuracy: On the KITTI benchmark (Moderate setting), RTS3D achieved a 10% improvement in Average Precision (AP) over previous image-only methods.
- Label Efficiency: The model achieves these results using only 3D bounding box labels for supervision, whereas competing models require depth maps, instance masks, and sometimes even CAD models.
Conclusion
RTS3D demonstrates that feature consistency between stereo views inherently encodes structural information, serving as an efficient proxy for depth. By detecting objects directly in the FCE space, the model avoids the bottleneck of generating standalone depth maps. While the model currently processes frames independently, future work will focus on temporal consistency and smoothing detection results between adjacent video frames to further improve stability.
Summary updated