TL;DR
This paper introduces RFM-Pose, a novel framework that significantly speeds up category-level 6D object pose estimation by combining flow matching for fast generation with reinforcement learning for refinement and selection. It achieves a better balance between speed and accuracy than previous methods, making real-time pose estimation more feasible.
What Problem Does This Paper Address?
Estimating the 6D pose (3D rotation and translation) of objects from images or point clouds is crucial for applications like robotics and virtual reality. A particularly challenging aspect is "category-level" pose estimation, where the system must identify the pose of objects belonging to a known category but which it hasn't seen before. This setting is difficult due to significant variations in object shape, appearance, and the inherent ambiguity caused by object symmetries, especially under occlusion or clutter. While recent generative models like diffusion models have shown promise in handling this ambiguity by generating multiple hypotheses, they suffer from high computational costs due to extensive iterative sampling and a separate evaluation step. This makes them impractical for real-time applications. RFM-Pose aims to bridge this gap by creating a faster and more efficient generative approach that can be directly optimized for pose accuracy.
Methods
RFM-Pose is built upon two core ideas: Flow Matching (FM) for efficient generation and Reinforcement Learning (RL) for refinement and selection.
- Flow Matching (FM) Backbone: Instead of diffusion models, RFM-Pose uses Flow Matching. FM learns a deterministic "flow field" that maps a simple prior distribution (like Gaussian noise) to the target data distribution (object poses). This is achieved by training a neural network to predict the velocity field along an optimal transport path. The advantage of FM is its significantly faster ODE-based generation, requiring fewer integration steps compared to diffusion models.
- Reinforcement Learning (RL) Refinement: The authors reframe the FM sampling process as a Markov Decision Process (MDP). Each step of integrating the flow field is treated as an "action." A pre-trained FM model serves as the initial "policy." This policy is then fine-tuned using Proximal Policy Optimization (PPO). The RL framework allows the model to learn from geometric error rewards, directly optimizing for accurate pose estimation.
- Multi-Critic Value Network: To handle the distinct nature of rotation and translation, a multi-critic value network is employed. This network estimates the expected return (reward) for both rotation and translation separately, providing dense learning signals to the policy.
- Value-Guided Pose Aggregation: At inference, RFM-Pose generates multiple pose candidates using the refined policy. Instead of simple averaging, these candidates are ranked and selected using the learned value network. This "value-guided" selection leverages the network's confidence assessments to pick the most promising hypotheses, which are then aggregated to produce the final pose.
Datasets
The paper evaluates RFM-Pose on three datasets:
- CAMERA: A synthetic dataset with 275K training and 25K testing images across six categories.
- REAL275: A real RGB-D dataset with 4.3K training images from 7 scenes and 2.75K testing images from 6 scenes, covering the same categories as CAMERA.
- Omni6DPose: A large-scale RGB-D dataset designed to provide broader object diversity for category-level pose estimation.
Key Insights and Ideas
- Flow Matching for Speed: The choice of Flow Matching over diffusion models is a key enabler for achieving real-time performance, as it drastically reduces the number of sampling steps required.
- RL for Direct Optimization: Reframing pose generation as an RL problem allows for direct optimization of pose accuracy, overcoming the limitation of generative models that are not directly trained with pose error feedback.
- Disentangled Critics for SE(3): Using separate critics for rotation and translation effectively addresses the challenges of optimizing for poses on the SE(3) manifold, which has distinct rotational and translational components.
- Value-Guided Selection: Leveraging the learned value network for selecting and aggregating pose candidates is more effective than simple averaging, especially for objects with symmetric geometries.
Key Results
- Speed-Accuracy Trade-off: RFM-Pose achieves a favorable speed-accuracy trade-off, outperforming GenPose on REAL275 with significantly fewer sampling steps (20 vs. 500) while achieving higher accuracy.
- Real-time Performance: The method operates above the real-time threshold of 30 FPS, making it practical for interactive applications.
- Improved Accuracy: On REAL275, RFM-Pose achieves 58.4% mAP (5°2cm metric) with only 20 sampling steps, outperforming previous work by 6.3% and using 25x fewer integration steps.
- Category-Level Tracking: The approach extends to category-level pose tracking, achieving 75.1% accuracy (5°5cm) at 34.5 FPS, which is 60% faster than GenPose.
- Generalization: RFM-Pose shows strong generalization capabilities on the Omni6DPose dataset, outperforming GenPose by a significant margin.
Limitations and Future Work
- Value Estimation Gap: While the value network performs well, there's still a gap between its performance and an oracle GT ranking, suggesting potential for further improvements in value estimation.
- Future Work: The authors suggest exploring RGB cues and broader object categories for future research.
Summary updated