Objectron: A Large Scale Dataset of Object-Centric Videos in the Wild With Pose Annotations

Adel Ahmadyan, Liangkai Zhang, Artsiom Ablavatski, Jianing Wei, Matthias Grundmann

Abstract

3D object detection has recently become popular due to many applications in robotics, augmented reality, autonomy, and image retrieval. We introduce the Objectron dataset to advance the state of the art in 3D object detection and foster new research and applications, such as 3D object tracking, view synthesis, and improved 3D shape representation. The dataset contains object-centric short videos with pose annotations for nine categories and includes 4 million annotated images in 14,819 annotated videos. We also propose a new evaluation metric, 3D Intersection over Union, for 3D object detection. We demonstrate the usefulness of our dataset in 3D object detection and novel view synthesis tasks by providing baseline models trained on this dataset. Our dataset and evaluation source code are available online at Github.com/google-research-datasets/Objectron.

Introduction

While 2D object detection has reached maturity with datasets like ImageNet and COCO, 3D object detection remains a significant challenge. Most existing 3D datasets are either synthetic, captured in controlled laboratory settings, or focused solely on autonomous driving (LiDAR-based). There is a significant gap in data for "everyday" 3D object understanding—essential for augmented reality (AR), robotics, and furniture placement applications. To understand an object in 3D, a model must observe it from multiple angles. Objectron addresses this by providing a massive, "in-the-wild" dataset of 15,000 object-centric videos featuring 4 million images with precise 3D bounding box annotations.

Methods

The researchers developed a specialized data collection and annotation pipeline to bridge the 2D-to-3D gap.

AR-Powered Collection and Annotation

Using mobile phones equipped with ARKit or ARCore, collectors recorded ~10-second videos of objects while moving around them. The AR software provided metadata including camera poses, sparse point clouds, and surface planes. Instead of annotating every frame, annotators placed a single 3D bounding box in a 3D world map, which was then projected back into every video frame to ensure temporal consistency.

Example of a 3D bounding box annotation on a book.

Architecture and Metrics

The authors proposed a baseline model using a two-stage architecture: a 2D detector (SSD) crops the object, followed by a second network (EfficientNet-Lite) that regresses 3D vertices. These points are "lifted" into 3D space using the EPnP algorithm. To evaluate performance, they implemented a robust 3D Intersection over Union (IoU) metric using the Sutherland-Hodgman polygon clipping algorithm to calculate the exact intersection volume of oriented 3D boxes.

The Objectron Dataset

The Objectron dataset contains 14,819 annotated videos (4M images) across nine categories: bikes, books, bottles, cameras, cereal boxes, chairs, cups, laptops, and shoes. It is geo-diverse, covering 10 countries across 5 continents.

The Objectron dataset features 9 categories of everyday objects with precise 3D pose annotations.

By using video rather than static images, the dataset captures multi-view geometry, mimicking how humans learn about objects. Because data was collected via AR sessions, object sizes and distances are stored in actual metric units. The dataset also addresses orientation ambiguity in symmetric objects (like bottles) by rotating the ground truth box during evaluation to find the maximum possible IoU.

Key Results

The proposed two-stage baseline (Crop + Regress) significantly outperformed single-stage models, achieving an average 3D IoU (at 0.5) of 0.6512. Analysis showed models were generally more accurate at estimating elevation than azimuth, likely due to a natural bias in collection (filming from a standing position). Furthermore, the high-quality camera poses in Objectron allow it to be used for Neural Radiance Fields (NeRF), enabling high-quality novel view synthesis.

Limitations and Future Work

Current limitations include the requirement for objects to remain stationary during capture to maintain AR tracking. There is also a viewpoint bias toward eye-level and top-down views. Future work will focus on expanding the dataset to hundreds of categories and capturing more diverse camera angles.

The Objectron dataset and its evaluation source code are available at github.com/google-research-datasets/Objectron.

Summary updated

Abstract

3D object detection has recently become popular due to many applications in robotics, augmented reality, autonomy, and image retrieval. We introduce the Objectron dataset to advance the state of the art in 3D object detection and foster new research and applications, such as 3D object tracking, view synthesis, and improved 3D shape representation. The dataset contains object-centric short videos with pose annotations for nine categories and includes 4 million annotated images in 14,819 annotated videos. We also propose a new evaluation metric, 3D Intersection over Union, for 3D object detection. We demonstrate the usefulness of our dataset in 3D object detection and novel view synthesis tasks by providing baseline models trained on this dataset. Our dataset and evaluation source code are available online at Github.com/google-research-datasets/Objectron.
BibTeX
@inproceedings{ahmadyan2021objectron,
  author = {Adel Ahmadyan and Liangkai Zhang and Artsiom Ablavatski and Jianing Wei and Matthias Grundmann},
  title = {{Objectron: A Large Scale Dataset of Object-Centric Videos in the Wild With Pose Annotations}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=U8sfBuKfdHe},
}
Figure