Introduction
In the field of robot learning, research typically prioritizes policy design—the "brain" of the robot. However, these policies are fundamentally limited by their "eyes," or observation space. While 2D RGB images are standard due to the availability of large-scale foundation models, they often fail to capture the 3D complexity of the physical world, leading to issues with depth perception and extreme sensitivity to environmental changes like lighting or camera placement.
This study introduces OBSBench, a comprehensive benchmark of 125 tasks designed to evaluate whether the choice of observation space (RGB, RGB-D, or Point Cloud) creates a performance ceiling for robotic agents.
Methodology and Framework
The researchers developed the OBSBench framework to facilitate a controlled comparison between different sensory modalities. The benchmark evaluates three primary observation spaces:
- RGB: Standard 2D color images.
- RGB-D: Color images augmented with a depth channel.
- Point Cloud (PC): Explicit 3D coordinates (x, y, z), often paired with color (r, g, b).
To process these inputs, the study utilizes a variety of encoders, including ResNet and Vision Transformers (ViT) for 2D data, and PointNet or SpUNet for 3D data. These are integrated with leading policy architectures such as the Action Chunking Transformer (ACT) and Diffusion Policy. The benchmark aggregates tasks from ManiSkill2 and RLBench, covering a wide range of rigid and soft-body manipulation scenarios.
Key Insights: The Superiority of 3D Geometry
The findings suggest that explicit 3D representation is a critical factor in robotic success:
- Geometric Grounding: Treating data as an explicit 3D structure (point cloud) significantly outperforms simply adding a depth channel to a 2D image (RGB-D). Geometry acts as a primary driver for success in manipulation.
- Robustness and Generalization: Point cloud models demonstrate remarkable resilience to camera shifts and lighting variations. While RGB-based models often fail when the camera viewpoint changes slightly, point cloud models maintain performance because the underlying geometry remains constant.
- Data Efficiency: 3D pre-training proves highly effective. For instance, the PonderV2 method, trained on only 4,500 3D samples, outperformed 2D models pre-trained on millions of images, suggesting that 3D representations are more information-dense for robotics.
Experimental Results
Quantitative analysis across OBSBench confirms the advantages of point cloud observations. In contact-rich tasks, point cloud methods achieved a mean success rate improvement of over 50% compared to their best 2D counterparts when using a Diffusion Policy.
Real-world validation on a physical bimanual robot mirrored these simulation results. In a "Reach Cube" task, the point cloud approach reached an 80% success rate, significantly higher than RGB (60%) and RGB-D (30%).
Conclusion and Future Directions
While point clouds offer superior robustness and generalization, they are not without challenges. Current 3D methods do not yet show a clear advantage in sample efficiency when training from scratch. Furthermore, the field lacks 3D datasets comparable in scale to 2D image repositories like ImageNet. Future research will likely focus on scaling 3D pre-training and developing foundation models specifically tailored for 3D robotic perception, potentially integrating additional modalities like tactile sensing to further bridge the gap between simulation and reality.
Summary updated