Any3D-VLA: Enhancing VLA Robustness via Diverse Point Clouds

Xianzhe Fan, Shengliang Deng, Xiaoyang Wu, Yuxiang Lu, Zhuoling Li, Mi Yan, Yujia Zhang, Zhizheng Zhang, He Wang, Hengshuang Zhao

TL;DR

Any3D-VLA enhances the spatial intelligence of Vision-Language-Action (VLA) models by integrating 3D point clouds into the decision-making loop. By training on a "hybrid" mix of simulated, sensor-based, and model-estimated 3D data, it achieves state-of-the-art robustness in real-world robot manipulation, even when high-quality depth sensors are unavailable.

What Problem Does This Paper Address?

Most current VLA models—the "brains" behind modern general-purpose robots—rely almost entirely on 2D image inputs. While 2D images are rich in semantic information (e.g., "this is a red mug"), they are inherently "flat." This makes robots brittle when faced with challenges like occlusions, small objects, or varying camera viewpoints where precise 3D spatial relationships are critical.

Adding 3D information seems like an obvious fix, but it faces three major hurdles:

  1. Data Scarcity: There is far less high-quality 3D robotic data compared to the massive 2D datasets used to train VLMs.
  2. The Domain Gap: Depth data from a simulator is "perfect," while data from a real-world sensor (like a RealSense camera) is noisy.
  3. Hardware Dependency: Many 3D methods require expensive depth sensors, making them hard to deploy on cheaper or specialized robot platforms.

Methods

The authors introduce Any3D-VLA, a modular "plug-in" architecture designed to inject 3D awareness into existing VLA backbones.

  • Lifting and Compression: The system takes RGB images and depth maps (either from a sensor or estimated by a model) and "lifts" them into a 3D point cloud. To keep the computation manageable, they use a voxel-grid sampling technique to compress tens of thousands of points into a few thousand representative ones.
  • 2D-3D Feature Fusion: Instead of replacing 2D features, Any3D-VLA uses a gated residual fusion mechanism. It aligns 3D point features with 2D image patches. The 3D information acts as a "correction" to the 2D semantics, providing the geometric "where" to the 2D "what."
  • Hybrid Point Cloud Training: This is the core innovation. During training, the model is exposed to a mix of ground-truth simulator depth and "noisy" depth estimated from single RGB images (using models like Depth Anything 3). This teaches the model to be domain-agnostic—it learns to extract useful geometric patterns regardless of whether the depth source is a high-end sensor or a software estimate.

Datasets

  • Synthetic RGBD Dataset: A large-scale dataset created using the Objaverse LVIS subset, featuring 290 object categories and over 10,000 instances in a cluttered tabletop simulator.
  • GRIT: A web-scale 2D grounding dataset used to maintain the model's general semantic understanding.
  • Real-World Post-Training: A small set of 100 demonstrations for specific tasks like "flower arrangement" and "cup carrier" to test fine-tuning capabilities.
  • Public Benchmarks: The model was also validated on the standard LIBERO and CALVIN robot manipulation benchmarks.

Key Insights and Ideas

  • Explicit 3D > Implicit 3D: The authors' pilot study found that simply adding a depth channel to a 2D CNN (implicit) isn't nearly as effective as actually building a 3D point cloud (explicit). The topological structure of 3D space is a powerful prior that 2D kernels struggle to replicate.
  • The "Correction" Philosophy: By using a gated fusion, the model doesn't lose the powerful pre-trained semantic knowledge of 2D vision transformers (like DINOv2). It only uses the 3D data to refine the spatial coordinates of its actions.
  • Robustness through Diversity: Training on "bad" model-estimated depth actually makes the model better at real-world deployment. It learns to ignore sensor artifacts and focus on the underlying geometry of the scene.

Key Results

  • Real-World Zero-Shot: In "out-of-the-box" testing (no real-world training), Any3D-VLA achieved a 62.5% success rate, outperforming the best 3D baseline (SpatialVLA) by 29.2%.
  • Robustness to Noise: The model maintained high performance even when using depth estimated from a single RGB image (no depth sensor at all), proving it can be deployed on robots with only standard cameras.
  • Fine-Tuning Excellence: After minimal post-training on real-world data, the model reached a 93.3% success rate on complex tasks like placing a flower in a vase.
  • Benchmark Performance: It improved over the GraspVLA baseline by 13.9% on the LIBERO benchmark.

Limitations and Future Work

While Any3D-VLA is a significant step forward, the authors note several areas for growth:

  • Scope: The current experiments are limited to a single robotic arm and a specific set of tabletop objects.
  • 3D Encoding: The 3D grid compression is relatively simple; more advanced 3D backbones could potentially extract even finer details.
  • Dynamics: The model currently focuses on static spatial understanding. Future work could incorporate temporal 3D information to better handle moving objects or dynamic environments.
  • Safety: As with all VLA models, the authors recommend human oversight and safety monitoring for real-world deployment, especially in assistive-care settings.

Summary updated

Abstract

Existing Vision-Language-Action (VLA) models typically take 2D images as visual input, which limits their spatial understanding in complex scenes. How can we incorporate 3D information to enhance VLA capabilities? We conduct a pilot study across different observation spaces and visual representations. The results show that explicitly lifting visual input into point clouds yields representations that better complement their corresponding 2D representations. To address the challenges of (1) scarce 3D data and (2) the domain gap induced by cross-environment differences and depth-scale biases, we propose Any3D-VLA. It unifies the simulator, sensor, and model-estimated point clouds within a training pipeline, constructs diverse inputs, and learns domain-agnostic 3D representations that are fused with the corresponding 2D representations. Simulation and real-world experiments demonstrate Any3D-VLA's advantages in improving performance and mitigating the domain gap. Our project homepage is available at https://xianzhefan.github.io/Any3D-VLA.github.io.
BibTeX
@inproceedings{fan2026any,
  author = {Xianzhe Fan and Shengliang Deng and Xiaoyang Wu and Yuxiang Lu and Zhuoling Li and Mi Yan and Yujia Zhang and Zhizheng Zhang and He Wang and Hengshuang Zhao},
  title = {{Any3D-VLA: Enhancing VLA Robustness via Diverse Point Clouds}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=zyMvoKYWMZ},
}
Figure