Object Reprojection Error (ORE): Camera pose benchmarks from lightweight tracking annotations

Xingyu Chen, Weiyao Wang, Hao Tang, Matt FeiszliNeurIPS 2023

Introduction

To build helpful AR assistants or robots, algorithms like Simultaneous Localization and Mapping (SLAM) or Structure from Motion (SfM) must accurately track a camera's movement through 3D space. Traditionally, evaluating these algorithms requires "ground truth" camera poses—data usually captured by bulky, expensive sensors like LiDAR or motion-capture systems.

Because of this hardware requirement, most benchmarks are limited to specific settings, such as self-driving cars on streets or robots in carefully scanned rooms. Consequently, the field lacks a robust way to measure how well these algorithms work "in the wild"—for instance, during everyday activities like cooking or gardening. These semantically rich videos rarely have 3D ground truth, leaving researchers in the dark about which tracking methods truly perform best in everyday scenarios.

The Object Reprojection Error (ORE) Metric

To bridge this gap, researchers introduce the Object Reprojection Error (ORE). Instead of comparing an estimated trajectory to a perfect sensor-derived trajectory, ORE uses static objects as landmarks, relying only on 2D bounding boxes.

Examples of ORE annotations in diverse egocentric videos

The ORE evaluation pipeline works as follows:

  1. Annotation: A video is annotated by drawing 2D bounding boxes around a few static objects (like a toaster or a picture frame) across several frames.
  2. Lifting to 3D: For a given camera trajectory being tested, the algorithm "lifts" a point from the center of a bounding box in one frame into 3D space. Because the true depth is unknown, an optimization step (grid search) finds the depth value that best fits the trajectory.
  3. Reprojection: That 3D point is then projected back onto all other frames using the estimated camera poses.
  4. Error Calculation: If the camera trajectory is accurate, the reprojected point should stay inside the object's bounding boxes throughout the video. If the point "drifts" outside the boxes, the trajectory is penalized.

This approach is highly scalable because drawing 2D bounding boxes is significantly easier and cheaper than capturing full 3D reconstructions. Furthermore, tracking objects rather than specific pixels or points is more robust against motion blur, lighting changes, and partial occlusions—common issues in head-mounted egocentric video.

The EgoStatic Benchmark

Using the ORE metric, the authors release EgoStatic, the largest benchmark to date for evaluating SLAM and SfM algorithms on diverse, real-world egocentric videos. Derived from the Ego4D and EgoTracks datasets, EgoStatic contains 5,708 video sequences (approximately 600 hours) and 22,000 static object tracklets. Covering 137 "everyday life" scenarios, it is significantly more diverse than previous benchmarks.

To validate ORE, the authors also tested it on ScanNet, a standard indoor dataset with ground truth 3D data. They found that ORE scores have a high rank correlation (Spearman Coefficient > 0.7) with traditional metrics like Absolute Trajectory Error (ATE). This confirms that if an algorithm performs better according to expensive LiDAR data, ORE will almost certainly reach the same conclusion using just bounding boxes.

Key Results

Evaluating state-of-the-art systems on EgoStatic revealed several critical insights:

  • In-the-Wild Tracking is Hard: Algorithms that perform nearly perfectly on indoor room scans (like DROID-SLAM) struggle significantly on the chaotic, shaky footage of EgoStatic.
  • Classic Methods Hold Their Own: In these wild scenarios, the classic geometry-based method COLMAP outperformed modern deep-learning-based methods. On EgoStatic, COLMAP achieved an ORE of 0.066, while the state-of-the-art DROID-SLAM scored 0.185 (lower is better).
  • Sensitivity: ORE is sensitive enough to help researchers tune hyperparameters, successfully detecting performance gains when adding steps like global bundle adjustment to a SLAM pipeline.

Limitations and Future Work

While powerful, ORE does have limitations. It relies heavily on the assumption that annotated objects are stationary. In highly dynamic or "social" scenarios (like a crowded party), finding enough truly static objects to calculate a reliable score is difficult. Additionally, relying on human labelers to determine if an object is static can introduce bias or error.

Despite these challenges, EgoStatic serves as a vital new playground for developing more robust SLAM systems capable of handling the unpredictable nature of human-worn camera footage.

Summary updated

BibTeX
@inproceedings{chen2023object,
  author = {Xingyu Chen and Weiyao Wang and Hao Tang and Matt Feiszli},
  title = {{Object Reprojection Error (ORE): Camera pose benchmarks from lightweight tracking annotations}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
}
Figure