OmniNOCS: A Unified NOCS Dataset and Model for 3D Lifting of 2D Objects

Akshay Krishnan, Abhijit Kundu, Kevis-Kokitsi Maninis, James Hays, Matthew Brown

TL;DR

This paper introduces OmniNOCS, a massive new dataset for 3D object understanding, and NOCSformer, a novel transformer-based model that can predict an object's 3D shape and pose from a single 2D image. This work significantly advances the ability of AI to understand and reconstruct 3D objects in diverse real-world scenarios.

What Problem Does This Paper Address?

Understanding the 3D world is crucial for many AI applications, from robotics and self-driving cars to augmented reality. A key challenge is accurately predicting an object's 6-Degree-of-Freedom (6DoF) pose (position and orientation) and its 3D shape from a single 2D image. Existing methods often struggle with generalization across a wide variety of object classes, environments, and camera setups. They are typically trained on limited datasets, making them brittle and unable to handle novel objects or scenarios. Furthermore, many existing datasets lack consistent object orientation annotations, which is critical for accurate 3D understanding.

Methods

The core of this work lies in two main contributions:

  • OmniNOCS Dataset: The authors introduce OmniNOCS, a large-scale dataset that significantly expands upon previous NOCS datasets. It includes NOCS maps, instance masks, and canonically oriented 3D bounding boxes for 97 object classes across indoor and outdoor scenes. This dataset is an order of magnitude larger than existing NOCS datasets in terms of both classes and instances.
  • NOCSformer Model: They propose NOCSformer, a novel transformer-based architecture for monocular NOCS prediction. NOCSformer takes a 2D object detection bounding box as input and predicts:
  • NOCS maps: These represent the 3D coordinates of points on an object's surface in a normalized, canonical space.
  • Instance masks: Precise segmentation of the object.
  • 3D bounding box: Including canonical orientation and metric size. The model leverages pre-trained Vision Transformers (ViTs) and a Dense Prediction Transformer (DPT) for feature extraction and fusion. A key aspect is its unified head for predicting NOCS and masks, and separate heads for size and pose estimation, all without class-specific parameters, enabling generalization.

Datasets

  • OmniNOCS: The primary contribution, a new large-scale dataset with NOCS annotations, instance masks, and canonical 3D bounding boxes for 97 object classes. It aggregates data from various sources, including KITTI, nuScenes, Objectron, ARKitScenes, Hypersim, SUN-RGBD, Cityscapes, Virtual KITTI, NOCS-Real275, and Waymo Open Dataset.
  • NOCS-Real275: Used for evaluating cross-dataset generalization.
  • nuScenes: Used for evaluating outdoor localization accuracy.
  • KITTI-val: Used for evaluating 3D orientation accuracy.
  • COCO: Used for qualitative evaluation of in-the-wild generalization.

Key Insights and Ideas

  • The Power of Scale and Diversity: The sheer scale and diversity of the OmniNOCS dataset are crucial for training a model that can generalize across many object classes and domains. This is a significant step beyond previous, more specialized datasets.
  • Canonical Orientation is Key: The paper emphasizes the importance of consistently oriented 3D bounding boxes. By carefully canonicalizing orientations across all constituent datasets, they create a more reliable foundation for 3D understanding.
  • Transformer Architecture for NOCS: Using a transformer-based architecture (NOCSformer) for NOCS prediction, rather than traditional CNNs, proves to be more scalable and effective for handling a large number of classes and generalizing to unseen data.
  • Unified Prediction Head: A single, large transformer head for predicting both NOCS and instance masks is more effective than separate heads, especially for a large-scale, diverse dataset.
  • Zero-Shot Generalization: NOCSformer demonstrates impressive zero-shot generalization capabilities, performing well on datasets and classes it was not explicitly trained on, highlighting the robustness of its learned representations.

Key Results

  • OmniNOCS Dataset: Contains over 380k images and 97 object categories, significantly larger than existing NOCS datasets.
  • NOCSformer Performance:
    • Achieves comparable 3D localization accuracy to state-of-the-art methods like Cube R-CNN on challenging benchmarks like nuScenes.
  • Outperforms existing NOCS baselines on the NOCS-Real275 dataset, even when NOCSformer is not trained on it (zero-shot evaluation).
  • Demonstrates strong generalization to unseen datasets and domains, including in-the-wild images from the web and COCO.
  • Achieves high 3D orientation accuracy, outperforming methods that use Chamfer loss.
  • Qualitative Results: The paper shows compelling qualitative results, demonstrating NOCSformer's ability to reconstruct 3D shapes and poses from single images across various scenes and object types.

Limitations and Future Work

  • Symmetric Objects and Reflected Geometry: The model struggles with symmetric objects (where multiple orientations are valid) and reflected geometry (e.g., left/right shoes). Future work could involve minimizing losses over multiple coordinate frames for symmetric objects or estimating left/right coordinate frames for reflected geometry.
  • Long-Range Accuracy: For small objects at very long distances, especially in single-camera applications, NOCSformer's accuracy deteriorates. This is attributed to the coarse input from small RoIs. The authors suggest that end-to-end models that leverage dataset biases might be more accurate in such scenarios.
  • Temporal Consistency: While the paper mentions temporal consistency in video sequences, it notes that this is achieved through independent frame predictions without explicit temporal smoothing. Future work could explore explicit temporal modeling.

Summary updated

Abstract

We propose OmniNOCS, a large-scale monocular dataset with 3D Normalized Object Coordinate Space (NOCS) maps, object masks, and 3D bounding box annotations for indoor and outdoor scenes. OmniNOCS has 20 times more object classes and 200 times more instances than existing NOCS datasets (NOCS-Real275, Wild6D). We use OmniNOCS to train a novel, transformer-based monocular NOCS prediction model (NOCSformer) that can predict accurate NOCS, instance masks and poses from 2D object detections across diverse classes. It is the first NOCS model that can generalize to a broad range of classes when prompted with 2D boxes. We evaluate our model on the task of 3D oriented bounding box prediction, where it achieves comparable results to state-of-the-art 3D detection methods such as Cube R-CNN. Unlike other 3D detection methods, our model also provides detailed and accurate 3D object shape and segmentation. We propose a novel benchmark for the task of NOCS prediction based on OmniNOCS, which we hope will serve as a useful baseline for future work in this area. Our dataset and code will be at the project website: https://omninocs.github.io.
BibTeX
@inproceedings{krishnan2024omninocs,
  author = {Akshay Krishnan and Abhijit Kundu and Kevis-Kokitsi Maninis and James Hays and Matthew Brown},
  title = {{OmniNOCS: A Unified NOCS Dataset and Model for 3D Lifting of 2D Objects}},
  booktitle = {European Conference on Computer Vision},
  year = {2024},
  doi = {10.1007/978-3-031-73226-3\_8},
}
Figure