RGB-D-Based Categorical Object Pose and Shape Estimation: Methods, Datasets, and Evaluation

Leonard Bruns, Patric Jensfelt

Introduction

In the field of robotics, the ability to accurately estimate an object's 6D pose and 3D shape is fundamental for autonomous interaction, such as grasping a tool or organizing a workspace. While "instance-level" estimation—where a precise 3D model of a specific object is known beforehand—has reached a high level of maturity, "category-level" estimation remains a significant challenge. This task requires models to generalize across a category (e.g., "mugs"), accounting for variations in geometry, scale, and appearance in objects they have never encountered during training.

This paper provides a critical "reality check" for the current state of categorical pose and shape estimation. The authors argue that while performance metrics on standard benchmarks appear high, these results are often inflated by dataset biases and flawed evaluation protocols. To address this, they introduce a more rigorous evaluation framework, a new "stress-test" dataset called REDWOOD75, and an open-source toolbox designed to standardize comparisons across the research community.

The Problem: Gaps in Current Research

The authors identify three primary obstacles hindering progress in the field:

  1. Evaluation Bias: The most widely used benchmark, REAL275, primarily features objects in upright positions on flat surfaces. This has led to models that rely on an "upright bias" rather than a true understanding of 3D geometry.
  2. Flawed Metrics: Traditional metrics like Average Precision (AP) and Chamfer Distance (CD) are often misleading. Chamfer Distance, in particular, is highly sensitive to point-cloud density and outliers, making it an unreliable measure of reconstruction quality.
  3. Lack of Standardization: Inconsistent training protocols and loss functions across different studies make it difficult to discern whether performance gains stem from architectural innovation or simply better hyperparameter tuning.

Methodology and Taxonomy

The paper categorizes existing estimation methods into two dominant families:

  • Correspondence-based Methods: Approaches such as SPD and SGPA attempt to establish a mapping between observed points and a "canonical" shape space (Normalized Object Coordinate Space, or NOCS). These methods typically rely on learning dense correspondences to resolve pose and shape.
  • Regression and Analysis-by-Synthesis: Methods like CASS or SDFEst either directly regress pose parameters from input data or iteratively optimize a shape representation (such as a Signed Distance Function) to align with the observation.

To improve evaluation reliability, the authors advocate for the use of Precision (at specific thresholds) and F-score. Unlike Chamfer Distance, the F-score measures the percentage of a reconstructed surface that falls within a specific distance (e.g., 1 cm) of the ground truth. This provides a more robust and physically meaningful metric for robotic applications where surface accuracy is paramount.

Datasets and the REDWOOD75 Stress Test

While the study reviews standard datasets like CAMERA25 (synthetic) and REAL275 (real-world, upright), its most significant contribution is the introduction of REDWOOD75.

REDWOOD75 consists of new annotations for the Redwood dataset, featuring categories like bottles, bowls, and mugs being freely rotated in human hands. Unlike tabletop datasets, REDWOOD75 presents objects in unconstrained, arbitrary orientations. This dataset serves as a "stress test" to determine if models can truly generalize or if they are simply exploiting the gravity-aligned priors of previous benchmarks.

Key Insights: The Upright Bias

The study's findings are revealing. Most state-of-the-art models fail significantly when faced with the unconstrained orientations in REDWOOD75. This confirms that many models are not learning the full 3D geometry of a category but are instead learning a strong prior that certain objects (like bottles) always stand vertically.

Interestingly, the authors found that depth-only methods (e.g., SDFEst and iCaps) tended to generalize better to unusual orientations than those relying heavily on RGB features. While depth-only methods can produce noisier outliers, they are less susceptible to the appearance-based biases inherent in RGB data.

Performance and Results

  • Standard Benchmarks: On the upright REAL275 dataset, DPDN and SGPA emerged as top performers in terms of precision.
  • Unconstrained Performance: On the REDWOOD75 stress test, SDFEst was the clear leader. It maintained a precision of approximately 0.64 (at 10°, 2cm), whereas most other models saw their performance plummet to near zero or below 0.20.
  • Efficiency Trade-offs: The authors highlight a clear "Pareto front" regarding speed and accuracy. For instance, CASS offers high-speed inference (~25ms) at the cost of reconstruction quality, while iCaps provides high accuracy but requires over 2 seconds per frame, making it unsuitable for real-time robotics.

Conclusion and Future Work

The paper concludes that the field must move beyond tabletop constraints to achieve true robustness. The authors emphasize the need for larger, high-quality datasets featuring unconstrained poses and suggest that future research should focus on:

  1. Sim-to-Real Generalization: Training on synthetic data that encompasses a full range of 3D rotations.
  2. Handling Ambiguity: Developing models capable of predicting multimodal distributions to account for symmetric objects where orientation is mathematically ambiguous.
  3. Computational Efficiency: Optimizing high-accuracy, unbiased methods like SDFEst for real-time deployment.

By providing a standardized toolbox and a more challenging benchmark, this work sets a new baseline for transparency and rigor in categorical object perception.

Summary updated

BibTeX
@article{bruns2023rgb,
  author = {Leonard Bruns and Patric Jensfelt},
  title = {{RGB-D-Based Categorical Object Pose and Shape Estimation: Methods, Datasets, and Evaluation}},
  journal = {Robotics Auton. Syst.},
  year = {2023},
  doi = {10.1016/j.robot.2023.104507},
}
Figure