Abstract
OmniD is a novel framework for robotic manipulation that constructs a 3D Bird's-Eye View (BEV) representation from multi-view RGB images, eliminating the need for expensive depth sensors or noisy point clouds. By integrating a deformable attention mechanism with a diffusion policy, OmniD significantly enhances a robot's ability to generalize to novel object positions, background textures, and camera viewpoints. This approach effectively bridges the gap between high-performing but brittle 2D policies and computationally intensive 3D methods.
Introduction
End-to-end visuomotor policies, such as Diffusion Policy or ACT, have advanced robot learning through human demonstrations. However, these image-based models often suffer from severe overfitting; minor changes in camera perspective or background textures can cause policy failure. While 3D representations like point clouds offer better spatial understanding, they typically require high-precision depth sensors and significant computational overhead. OmniD addresses this by asking: can we achieve robust 3D spatial understanding and out-of-distribution (OOD) generalization using only standard, lightweight 2D multi-view cameras?
Methods
The Omni-Vision Diffusion Policy (OmniD) processes multi-view camera feeds through a three-stage pipeline:
- 2D Feature Extraction: Multi-view RGB images are processed by a shared ResNet-18 backbone to extract spatial features.
- Omni 3D Feature Generator (OFG): This core component uses deformable attention to fuse 2D features into a unified 3D Bird's-Eye View (BEV) voxel grid.
- Diffusion Policy Head: Compressed 3D features and robot proprioception are fed into a diffusion head to iteratively denoise and predict smooth robot actions.
Omni 3D Feature Generator (OFG)
The OFG is the primary innovation of OmniD. Inspired by autonomous driving architectures like BEVFormer, it employs deformable attention to map 2D features into 3D space. Instead of uniform processing, the network learns spatial queries and sampling offsets, allowing it to focus on task-relevant areas (such as the object being manipulated) while ignoring irrelevant background pixels.
Key Insights
- BEV for Manipulation: Applying Bird's-Eye View representations to robotic manipulation provides strong spatial priors without the noise and computational bloat of point clouds.
- Generalization via Attention: Deformable attention acts as a filter, enabling the model to ignore background shifts that typically cause 2D policies to fail.
- Efficient Adaptation: The robust 3D feature space allows for rapid fine-tuning to novel camera viewpoints with as few as 10 human demonstrations.
Experimental Results
OmniD was evaluated in both simulation (VLABench) and real-world settings, demonstrating significant improvements over baselines:
- In-Distribution (ID) Performance: In simulation, OmniD achieved a 91.0% success rate, outperforming Diffusion Policy (82.0%) and ACT (43.3%). Real-world ID success reached 84%.
- Out-of-Distribution (OOD) Robustness: OmniD maintained success rates up to 90% on novel backgrounds where baselines failed completely (0%). It also showed superior performance on novel object positions.
- Few-Shot Adaptation: When fine-tuned on a new camera viewpoint with only 10 trajectories, OmniD achieved an 88% success rate in simulation and 60% in the real world, while baseline models failed to adapt entirely.
Conclusion and Future Work
OmniD demonstrates that 3D spatial understanding can be effectively derived from 2D images for robust robot manipulation. While the model currently requires approximate camera extrinsics, it eliminates the dependency on depth sensors. Future work will explore multi-task learning for long-horizon tasks and addressing "Combinatorial-Distribution" scenarios where multiple OOD shifts occur simultaneously.
Summary updated