NeurOCS: Neural NOCS Supervision for Monocular 3D Object Localization

Zhixiang Min, Bingbing Zhuang, Samuel Schulter, Buyu Liu, Enrique Dunn, Manmohan Chandraker

Abstract

Monocular 3D object localization remains a fundamental challenge in computer vision due to the inherent depth ambiguity of single-view images. While Normalized Object Coordinate Space (NOCS) provides a robust geometric framework for pose estimation via the Perspective-n-Point (PnP) algorithm, obtaining high-quality NOCS labels for real-world driving scenarios is notoriously difficult. This paper presents NeurOCS, a framework that leverages Neural Radiance Fields (NeRF) to provide dense, self-supervised NOCS labels. By training a deformable categorical shape model on real-world data using only 2D masks and 3D boxes, NeurOCS generates high-fidelity pseudo-ground truth for training a standard regressor. The resulting system achieves state-of-the-art performance on the KITTI benchmark, demonstrating the power of neural rendering as a supervisory signal for geometric tasks.

Introduction

Estimating the 3D position, orientation, and dimensions of objects from a single image—monocular 3D localization—is a critical capability for low-cost autonomous driving systems. However, the task is mathematically ill-posed because a single 2D projection can correspond to infinitely many 3D configurations.

A promising approach to resolve this ambiguity involves predicting the Normalized Object Coordinate Space (NOCS) for every visible pixel. NOCS maps relate 2D image coordinates to a canonical 3D coordinate system defined for an object category. Once these correspondences are established, the object's pose can be recovered using a PnP solver. The primary bottleneck for this approach in outdoor environments is the lack of supervision. Unlike indoor datasets where CAD models provide perfect ground truth, real-world driving datasets like KITTI offer only sparse Lidar points or coarse bounding boxes, which are insufficient for training dense NOCS regressors.

The NeurOCS Framework

NeurOCS addresses the supervision gap through a "Teacher-Student" architecture that utilizes differentiable rendering to bridge 2D observations and 3D labels.

1. The Teacher: Deformable Categorical Shape Model

The core of the framework is a NeRF-based shape representation that learns a category-level prior. Instead of a static model, the authors employ a low-rank deformable latent grid. This allows the model to represent a "canonical" car shape while accommodating instance-specific variations, such as the difference between a hatchback and an SUV. This teacher model is trained using differentiable rendering, supervised by 2D instance masks and 3D bounding boxes. By optimizing the shape to match the 2D projections across the training set, the model learns to reconstruct dense 3D geometry from sparse inputs.

2. The Student: NOCS Regressor

Once the categorical shape model is trained, it is used to render dense, noise-free NOCS maps for the training images. These rendered maps serve as high-quality pseudo-ground truth. A standard ResNet-50 based "student" network is then trained to predict these NOCS maps directly from image crops. A key advantage of this approach is inference efficiency: the complex NeRF component is only required during the training phase. At test time, the system relies solely on the fast NOCS regressor and a PnP solver.

3. Object-Centric Learning and Scale Fusion

NeurOCS adopts an "object-centric" approach, training on cropped images of vehicles with the background removed. This design choice prevents the network from relying on environmental context—such as the position of the road—and forces it to learn the intrinsic 3D geometry of the vehicle. To further improve robustness, the system employs "Scale Fusion," combining the depth calculated from the PnP solver with a direct depth prediction from a base 3D detector to mitigate errors in distant object localization.

Experimental Results

NeurOCS was primarily evaluated on the KITTI-Object benchmark, focusing on the "Car" category, and tested for generalization on the Waymo Open and NuScenes datasets.

Benchmark Performance

At the time of its release, NeurOCS achieved the top position on the KITTI monocular 3D detection leaderboard. In the "Easy" category, it reached a 29.89% $AP_{40}$ for 3D localization. This represents a significant improvement over previous geometric methods like MonoRUn (19.65%) and hybrid approaches like MonoDDE (24.93%).

Key Insights

  • NeRF as a Labeler: The study demonstrates that differentiable rendering is more effective as a tool for generating training data than as a test-time optimization loop. This keeps the final model fast enough for real-time applications.
  • The Value of Priors: By learning what a car "generally" looks like, the model successfully fills in geometric gaps where Lidar data is sparse or missing.
  • Context Bias: The finding that removing scene context improves performance suggests that many existing monocular methods may be over-fitting to specific road geometries rather than truly understanding 3D object shape.

Conclusion and Future Work

NeurOCS provides a powerful methodology for supervising 3D tasks without requiring expensive 3D annotations or CAD models. While the system currently relies on a base detector for initial object proposals and may struggle with highly irregular vehicle shapes (e.g., specialized construction equipment), it sets a new standard for monocular localization. Future research directions include the end-to-end integration of the NOCS regressor with the base detector and expanding the framework to a wider variety of object categories.

Summary updated

Abstract

Monocular 3D object localization in driving scenes is a crucial task, but challenging due to its ill-posed nature. Estimating 3D coordinates for each pixel on the object surface holds great potential as it provides dense 2D-3D geometric constraints for the underlying PnP problem. However, high-quality ground truth supervision is not available in driving scenes due to sparsity and various artifacts of Lidar data, as well as the practical infeasibility of collecting per-instance CAD models. In this work, we present NeurOCS, a framework that uses instance masks and 3D boxes as input to learn 3D object shapes by means of differentiable rendering, which further serves as supervision for learning dense object coordinates. Our approach rests on insights in learning a category-level shape prior directly from real driving scenes, while properly handling single-view ambiguities. Furthermore, we study and make critical design choices to learn object coordinates more effectively from an object-centric view. Altogether, our framework leads to new state-of-the-art in monocular 3D localization that ranks 1st on the KITTI-Object benchmark among published monocular methods.
BibTeX
@inproceedings{min2023neurocs,
  author = {Zhixiang Min and Bingbing Zhuang and Samuel Schulter and Buyu Liu and Enrique Dunn and Manmohan Chandraker},
  title = {{NeurOCS: Neural NOCS Supervision for Monocular 3D Object Localization}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
}
Figure