Objectron: A Large Scale Dataset of Object-Centric Videos in the Wild with Pose Annotations

Adel Ahmadyan, Liangkai Zhang, Jianing Wei, Artsiom Ablavatski, Matthias Grundmann

Abstract

3D object detection has recently become popular due to many applications in robotics, augmented reality, autonomy, and image retrieval. We introduce the Objectron dataset to advance the state of the art in 3D object detection and foster new research and applications, such as 3D object tracking, view synthesis, and improved 3D shape representation. The dataset contains object-centric short videos with pose annotations for nine categories and includes 4 million annotated images in 14,819 annotated videos. We also propose a new evaluation metric, 3D Intersection over Union, for 3D object detection. We demonstrate the usefulness of our dataset in 3D object detection tasks by providing baseline models trained on this dataset. Our dataset and evaluation source code are available online at http://www.objectron.dev

Introduction

While 2D object detection has achieved significant milestones through datasets like ImageNet and COCO, 3D object detection remains a complex frontier. Understanding an object's precise position, orientation, and physical dimensions in 3D space is vital for the advancement of robotics, autonomous navigation, and augmented reality (AR). Historically, the development of robust 3D models has been hindered by data limitations. Existing datasets are frequently synthetic, lacking real-world complexity; controlled, filmed in static laboratory settings; or scene-centric, focusing on entire environments rather than individual objects.

The Objectron dataset addresses these gaps by providing a massive collection of "in the wild" video data. It features common objects recorded in diverse, real-world settings across ten countries, moving the field beyond the constraints of synthetic and laboratory data.

Methodology

To scale 3D annotation efficiently, the researchers leveraged mobile augmented reality (AR) metadata. By using smartphones equipped with ARKit or ARCore, they obtained camera poses, sparse point clouds, and surface planes as part of the data collection process.

3D bounding box annotations projected onto video frames.

Annotation Process

Instead of manual frame-by-frame annotation, which is prohibitively expensive, annotators define a 3D bounding box within a 3D world map generated from the AR session. This box is then projected back onto every frame of the video, ensuring perfect temporal consistency and significantly reducing the cost per annotated image.

3D Intersection over Union (IoU)

Standard 3D IoU metrics often assume axis-aligned boxes. Objectron introduces a more robust algorithm for general, oriented 3D boxes. This approach utilizes the Sutherland-Hodgman polygon clipping algorithm to calculate the overlap between two arbitrary oriented boxes, providing a more accurate measure for 3D detection performance.

The Objectron Dataset

The Objectron dataset consists of 14,819 annotated videos and 4 million annotated images. It covers nine common object classes: bikes, books, bottles, cameras, cereal boxes, chairs, cups, laptops, and shoes.

Examples of the nine object categories in the Objectron dataset with 3D bounding box annotations.

The data is characterized by its high diversity, with samples collected from five continents. Compared to existing benchmarks like LineMOD or YCB, Objectron is substantially larger and provides higher-resolution video captured in unconstrained environments.

Key Insights and Evaluation

A core philosophy of the dataset is that "movement is perception." By providing video sequences, Objectron allows models to learn 3D structures by observing objects from multiple viewpoints. Furthermore, because the data is collected via AR sessions, annotations are provided in metric scale, enabling models to learn the actual physical size of objects.

The evaluation framework also addresses the "Symmetry Problem." For objects like bottles or cups that appear identical when rotated around their central axis, the new metric accounts for this symmetry to avoid unfairly penalizing models for visually indistinguishable rotation errors.

Results and Baselines

The researchers established two baseline models:

  1. MobilePose: A lightweight, single-stage network optimized for real-time mobile applications.
  2. Two-Stage Model: A robust architecture that uses a Single Shot Detector (SSD) for object cropping followed by an EfficientNet-Lite regressor to predict 3D keypoints.

The Two-Stage model demonstrated superior performance, achieving an Average Precision (AP) of 0.8505 at 0.5 3D IoU for the "Chair" category, compared to 0.7112 for MobilePose. Analysis showed that models were generally more accurate at estimating elevation than azimuth, likely due to the natural tendency of human collectors to record objects from a downward angle.

Conclusion and Future Work

While Objectron represents a significant leap forward, challenges remain. The current dataset has a viewpoint bias toward top-down perspectives, and the number of categories (nine) is small compared to 2D counterparts. Future work will involve expanding the category count and addressing the fundamental challenge of rotation estimation for perfectly symmetrical objects. The dataset and evaluation tools are publicly available at http://www.objectron.dev.

Summary updated

Abstract

3D object detection has recently become popular due to many applications in robotics, augmented reality, autonomy, and image retrieval. We introduce the Objectron dataset to advance the state of the art in 3D object detection and foster new research and applications, such as 3D object tracking, view synthesis, and improved 3D shape representation. The dataset contains object-centric short videos with pose annotations for nine categories and includes 4 million annotated images in 14,819 annotated videos. We also propose a new evaluation metric, 3D Intersection over Union, for 3D object detection. We demonstrate the usefulness of our dataset in 3D object detection tasks by providing baseline models trained on this dataset. Our dataset and evaluation source code are available online at http://www.objectron.dev
BibTeX
@inproceedings{ahmadyan2021objectron,
  author = {Adel Ahmadyan and Liangkai Zhang and Jianing Wei and Artsiom Ablavatski and Matthias Grundmann},
  title = {{Objectron: A Large Scale Dataset of Object-Centric Videos in the Wild with Pose Annotations}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=U8sfBuKfdHe},
}
Figure