TL;DR
Any3D-VLA enhances the spatial intelligence of Vision-Language-Action (VLA) models by integrating 3D point clouds into the decision-making loop. By training on a "hybrid" mix of simulated, sensor-based, and model-estimated 3D data, it achieves state-of-the-art robustness in real-world robot manipulation, even when high-quality depth sensors are unavailable.
What Problem Does This Paper Address?
Most current VLA models—the "brains" behind modern general-purpose robots—rely almost entirely on 2D image inputs. While 2D images are rich in semantic information (e.g., "this is a red mug"), they are inherently "flat." This makes robots brittle when faced with challenges like occlusions, small objects, or varying camera viewpoints where precise 3D spatial relationships are critical.
Adding 3D information seems like an obvious fix, but it faces three major hurdles:
- Data Scarcity: There is far less high-quality 3D robotic data compared to the massive 2D datasets used to train VLMs.
- The Domain Gap: Depth data from a simulator is "perfect," while data from a real-world sensor (like a RealSense camera) is noisy.
- Hardware Dependency: Many 3D methods require expensive depth sensors, making them hard to deploy on cheaper or specialized robot platforms.
Methods
The authors introduce Any3D-VLA, a modular "plug-in" architecture designed to inject 3D awareness into existing VLA backbones.
- Lifting and Compression: The system takes RGB images and depth maps (either from a sensor or estimated by a model) and "lifts" them into a 3D point cloud. To keep the computation manageable, they use a voxel-grid sampling technique to compress tens of thousands of points into a few thousand representative ones.
- 2D-3D Feature Fusion: Instead of replacing 2D features, Any3D-VLA uses a gated residual fusion mechanism. It aligns 3D point features with 2D image patches. The 3D information acts as a "correction" to the 2D semantics, providing the geometric "where" to the 2D "what."
- Hybrid Point Cloud Training: This is the core innovation. During training, the model is exposed to a mix of ground-truth simulator depth and "noisy" depth estimated from single RGB images (using models like Depth Anything 3). This teaches the model to be domain-agnostic—it learns to extract useful geometric patterns regardless of whether the depth source is a high-end sensor or a software estimate.
Datasets
- Synthetic RGBD Dataset: A large-scale dataset created using the Objaverse LVIS subset, featuring 290 object categories and over 10,000 instances in a cluttered tabletop simulator.
- GRIT: A web-scale 2D grounding dataset used to maintain the model's general semantic understanding.
- Real-World Post-Training: A small set of 100 demonstrations for specific tasks like "flower arrangement" and "cup carrier" to test fine-tuning capabilities.
- Public Benchmarks: The model was also validated on the standard LIBERO and CALVIN robot manipulation benchmarks.
Key Insights and Ideas
- Explicit 3D > Implicit 3D: The authors' pilot study found that simply adding a depth channel to a 2D CNN (implicit) isn't nearly as effective as actually building a 3D point cloud (explicit). The topological structure of 3D space is a powerful prior that 2D kernels struggle to replicate.
- The "Correction" Philosophy: By using a gated fusion, the model doesn't lose the powerful pre-trained semantic knowledge of 2D vision transformers (like DINOv2). It only uses the 3D data to refine the spatial coordinates of its actions.
- Robustness through Diversity: Training on "bad" model-estimated depth actually makes the model better at real-world deployment. It learns to ignore sensor artifacts and focus on the underlying geometry of the scene.
Key Results
- Real-World Zero-Shot: In "out-of-the-box" testing (no real-world training), Any3D-VLA achieved a 62.5% success rate, outperforming the best 3D baseline (SpatialVLA) by 29.2%.
- Robustness to Noise: The model maintained high performance even when using depth estimated from a single RGB image (no depth sensor at all), proving it can be deployed on robots with only standard cameras.
- Fine-Tuning Excellence: After minimal post-training on real-world data, the model reached a 93.3% success rate on complex tasks like placing a flower in a vase.
- Benchmark Performance: It improved over the GraspVLA baseline by 13.9% on the LIBERO benchmark.
Limitations and Future Work
While Any3D-VLA is a significant step forward, the authors note several areas for growth:
- Scope: The current experiments are limited to a single robotic arm and a specific set of tabletop objects.
- 3D Encoding: The 3D grid compression is relatively simple; more advanced 3D backbones could potentially extract even finer details.
- Dynamics: The model currently focuses on static spatial understanding. Future work could incorporate temporal 3D information to better handle moving objects or dynamic environments.
- Safety: As with all VLA models, the authors recommend human oversight and safety monitoring for real-world deployment, especially in assistive-care settings.
Summary updated