Recursive Cross-View: Use Only 2D Detectors to Achieve 3D Object Detection without 3D Annotations

Shun Gui, Yan Luximon

TL;DR

Researchers have developed Recursive Cross-View (RCV), a method that achieves fully oriented 3D object detection using only standard 2D detectors and zero 3D annotations. By recursively projecting point clouds into 2D "pseudo-views" and refining the results, RCV outperforms existing image-based methods and can even act as an automated 3D data annotator.

What Problem Does This Paper Address?

3D object detection is essential for robots and self-driving cars to understand their surroundings. However, training these systems usually requires massive amounts of 3D bounding box data—labels that are notoriously difficult and expensive to produce. For context, the authors note that annotating the SUN RGB-D dataset required over 2,000 hours of manual labor.

Furthermore, many existing "weakly supervised" methods are limited to objects with vertical orientations (like cars on a flat road). If you want to detect a hand or a tool tilted at an odd angle, these systems often fail. The gap RCV fills is the need for a system that is data-efficient (no 3D labels), flexible (handles any orientation), and fast to deploy on new types of objects.

Methods

The core philosophy of RCV is inspired by traditional engineering drawings: a 3D object can be perfectly described by its front, top, and side views.

The RCV pipeline follows a three-step recursive process:

  1. Perspective View (The Proposal): It starts with a standard 2D detector (like YOLOv5) running on an RGB image. This 2D box is projected into 3D space to create a "frustum"—a 3D cone-shaped slice of the point cloud where the object likely resides.
  2. Recursive Orthographic Cross-View (The Refinement): This is the "secret sauce." The system projects the points inside the frustum onto two orthogonal planes to create "pseudo-view" images. It then runs the 2D detector on these pseudo-views to crop out background noise and tighten the 3D box.
  3. Convergence: This process repeats. With each iteration, the projection axes align more closely with the object's true orientation, and the 3D box becomes more precise. The recursion stops when the box dimensions stabilize.

Crucially, the system only needs 2D labels to train the underlying detector, which are much easier to obtain than 3D boxes.

Datasets

The authors evaluated RCV on two major benchmarks and two custom real-world scenarios:

  • SUN RGB-D: An indoor benchmark. RCV was tested on 10 object categories.
  • KITTI: A famous autonomous driving dataset. RCV was tested specifically on pedestrians and cyclists to prove data efficiency.
  • 3D HUMAN & 3D HAND (New): The authors collected their own data using an Azure Kinect sensor to demonstrate how RCV can automatically generate 3D annotations for new tasks.

Key Insights and Ideas

  • 2D Detectors are "3D Aware": The "aha" moment here is that a mature 2D detector, when shown a projection of a point cloud from the right angle, is remarkably good at identifying object boundaries. By treating the 3D detection problem as a series of 2D tasks, we can leverage the massive progress made in 2D computer vision.
  • Orientation via Eigenvectors: To handle "fully oriented" boxes (objects tilted in any direction), RCV uses the eigenvectors of the point cloud to estimate the initial projection axes, which are then refined through the recursive loop.
  • The "Automatic Annotator": Because RCV doesn't need 3D labels to work, it can be used to "pre-label" new datasets. A human only needs to provide a few 2D boxes, and RCV generates the 3D boxes, which can then be used to train other, more complex models.

Key Results

  • Superior Performance: On the SUN RGB-D benchmark, RCV achieved a mean Average Precision (mAP) of 44.95, outperforming state-of-the-art monocular methods like ImVoxelNet (43.66) and T3DU (26.38).
  • Data Efficiency: On the KITTI dataset, RCV trained on only 25% of the available data still significantly outperformed existing methods that used 100% of the data.
  • Real-Time Capability: When deployed on a live RGB-D stream, the system achieved a processing speed of 7 FPS, making it viable for interactive robotic applications.

Limitations and Future Work

While RCV is a major step forward for "label-free" 3D detection, the authors acknowledge a few hurdles:

  • Dependency on 2D Quality: If the initial 2D detector fails or provides a poor confidence score, the recursion may never start or may converge on the wrong area.
  • Large Object Struggles: The system performs relatively worse on very large objects. The authors hypothesize this is because cameras often only capture a small portion of a large object, making it harder for the "three-view principle" to reconstruct the full geometry.
  • Strategy Exploration: Currently, the team has only explored one "divide-and-conquer" recursive strategy; they suggest that other recursive logic might yield even better stability or speed.

Summary updated

Abstract

Heavily relying on 3D annotations limits the real-world application of 3D object detection. In this paper, we propose a method that does not demand any 3D annotation, while being able to predict fully oriented 3D bounding boxes. Our method, called Recursive Cross-View (RCV), utilizes the three-view principle to convert 3D detection into multiple 2D detection tasks, requiring only a subset of 2D labels. We propose a recursive paradigm, in which instance segmentation and 3D bounding box generation by Cross-View are implemented recursively until convergence. Specifically, our proposed method involves the use of a frustum for each 2D bounding box, which is then followed by the recursive paradigm that ultimately generates a fully oriented 3D box, along with its corresponding class and score. Note that, class and score are given by the 2D detector. Estimated on the SUN RGB-D and KITTI datasets, our method outperforms existing image-based approaches. To justify that our method can be quickly used to new tasks, we implement it on two real-world scenarios, namely 3D human detection and 3D hand detection. As a result, two new 3D annotated datasets are obtained, which means that RCV can be viewed as a (semi-) automatic 3D annotator. Furthermore, we deploy RCV on a depth sensor, which achieves detection at 7 fps on a live RGB-D stream. RCV is the first 3D detection method that yields fully oriented 3D boxes without consuming 3D labels.
BibTeX
@article{gui2023recursive,
  author = {Shun Gui and Yan Luximon},
  title = {{Recursive Cross-View: Use Only 2D Detectors to Achieve 3D Object Detection without 3D Annotations}},
  journal = {IEEE Robotics and Automation Letters},
  year = {2023},
  doi = {10.1109/lra.2023.3307282},
}
Figure