CPPF++: Uncertainty-Aware Sim2Real Object Pose Estimation by Vote Aggregation

Yang You, Wenhao He, Jin Liu, Hongkai Xiong, Weiming Wang, Cewu LuIEEE TPAMI 2024 · DOI

TL;DR

This paper introduces CPPF++, a novel method for object pose estimation that bridges the gap between synthetic training data and real-world applications. By modeling voting uncertainty and incorporating N-point tuples for richer context, CPPF++ achieves state-of-the-art performance in sim-to-real transfer and introduces a challenging new dataset, DiversePose 300, to further advance the field.

What Problem Does This Paper Address?

Object pose estimation, determining an object's 6D pose (3D position and 3D orientation) from an image, is crucial for many computer vision tasks like robotic manipulation and augmented reality. While current methods achieve impressive results, they often rely on large datasets of real-world annotated images, which are expensive and time-consuming to acquire. This limits their generalizability. To overcome this, researchers have explored training models on synthetic data generated from 3D CAD models. However, the "sim-to-real" gap – the performance drop when a model trained on synthetic data is applied to real-world images – remains a significant challenge. Existing sim-to-real methods often struggle with background noise, occlusions, and the inherent differences between synthetic and real data. This paper specifically targets the "pure sim-to-real" setting, where models are trained only on synthetic data without any real-world background information, aiming for practical and robust pose estimation without the need for real-world annotations.

Methods

CPPF++ builds upon the previous CPPF method but introduces several key innovations:

  • Uncertainty-Aware Vote Aggregation: Instead of directly predicting deterministic voting targets (like object center and orientation), CPPF++ models these as probability distributions. This probabilistic approach helps to mitigate "voting collision," where different point pairs with similar features lead to conflicting votes.
  • N-Point Tuples: To capture richer contextual information than simple point pairs, the method uses N-point tuples (e.g., 3-point tuples). These tuples provide more robust features that are less susceptible to noise and better at differentiating between similar geometric configurations.
  • Noisy Pair Filtering: A module is introduced to filter out unreliable point pairs that might originate from background noise or occlusions, ensuring that only high-quality votes contribute to the final pose estimation.
  • Importance Sample Re-weighting: After noisy pair filtering, some points might have fewer associated reliable pairs. This module re-weights points to ensure each point contributes equally to the final pose, preventing bias.
  • Online Alignment Optimization: A differentiable optimization step is applied during inference to further refine the predicted rotation and translation, improving accuracy.
  • Tuple Feature Ensemble: The method combines both geometric (SHOT descriptor) and visual (DINOv2 features) cues from N-point tuples, recognizing that different objects might rely more heavily on one type of information than the other.

Datasets

  • NOCS REAL275: A benchmark dataset for category-level pose estimation, used for evaluating the method's performance.
  • Wild6D, PhoCAL: Existing datasets used to evaluate the generalizability of the proposed method to "in-the-wild" scenarios.
  • DiversePose 300 (New Dataset): Introduced by the authors, this dataset is designed to be more challenging for category-level pose estimation. It features a wide variety of poses and background distributions across three object categories (bowls, bottles, mugs) and is organized into three difficulty levels (Easy, Medium, Hard).

Key Insights and Ideas

  • Modeling Uncertainty is Crucial: Directly predicting deterministic voting targets leads to performance degradation due to voting collision. Probabilistically modeling these targets significantly improves robustness.
  • Context is King (Even in Sim-to-Real): Simple point pairs lack sufficient context. Using N-point tuples provides richer information, enabling better differentiation and reducing the impact of noise.
  • Pure Sim-to-Real is Achievable and Practical: The paper demonstrates that training solely on synthetic data without backgrounds can yield competitive or superior results compared to methods that use real-world data or synthetic data with simulated backgrounds.
  • Diverse Datasets are Needed: Existing datasets, even those with real-world data, may not fully capture the complexity of real-world scenarios. The DiversePose 300 dataset aims to fill this gap by offering greater pose and background variation.

Key Results

  • CPPF++ significantly outperforms previous sim-to-real methods on the NOCS REAL275 dataset, achieving comparable or superior performance to methods trained on real-world data.
  • The method demonstrates strong generalizability to unseen datasets like Wild6D, PhoCAL, and the newly introduced DiversePose 300, outperforming baseline methods that tend to overfit to their training data.
  • Ablation studies show that each proposed module (uncertainty modeling, N-point tuples, noisy pair filtering, re-weighting, and online alignment) contributes to the overall performance improvement.
  • The DiversePose 300 dataset presents a significant challenge, with even state-of-the-art methods struggling to achieve high performance, highlighting the need for more robust approaches.

Limitations and Future Work

  • Camera Category Performance: The method shows a performance drop on the "camera" category in NOCS REAL275. This is attributed to a significant disparity between the synthetic CAD models and real-world camera objects, suggesting that further improvements in synthetic data realism or domain adaptation might be needed.
  • Instance Segmentation Dependency: While the method aims for pure sim-to-real, it still relies on instance segmentation to sample points. Although they explore zero-shot segmentation, the accuracy of this initial step can impact overall performance, as seen with the "pitcher_base" failure case on YCB-Video when using a less robust detector.
  • Feature Extraction Bottleneck: The generation of DINOv2 features during data preparation is currently a significant bottleneck. Future work could focus on developing more lightweight DINO-like feature extractors to speed up training.
  • Further Dataset Exploration: The authors suggest that the DiversePose 300 dataset can serve as a valuable resource for future research in object pose estimation.

Summary updated

BibTeX
@article{you2024cppf,
  author = {Yang You and Wenhao He and Jin Liu and Hongkai Xiong and Weiming Wang and Cewu Lu},
  title = {{CPPF++: Uncertainty-Aware Sim2Real Object Pose Estimation by Vote Aggregation}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2024},
  doi = {10.1109/tpami.2024.3419038},
}
Figure