TL;DR
GaussianWAM is a training-time enhancement framework that injects 3D geometric and semantic awareness into World-Action Models (WAMs) for robotic manipulation. By organizing heterogeneous teacher signals within a unified 3D Gaussian field during training, it significantly boosts policy robustness against visual and environmental shifts. Crucially, all 3D components are discarded before deployment, delivering these performance gains with zero additional inference overhead.
What Problem Does This Paper Address?
World-Action Models (WAMs) represent a powerful paradigm in robotics, combining future video prediction with action generation to learn implicit physical and temporal dynamics. However, because these models primarily operate within 2D video or image latent spaces, their representations are not explicitly constrained to preserve cross-view 3D geometric structures or localized object semantics.
While one could simply distill features from pretrained vision and geometry foundation models directly into the WAM, doing so treats geometry and semantics as disconnected 2D signals. Conversely, explicitly building 3D or 4D world models typically requires heavy architectural modifications, such as 3D encoders, decoders, or rendering modules, which drastically increase computational costs during deployment.
Methods
The authors introduce GaussianWAM, a framework that leverages a 3D Gaussian field as a unified spatial carrier strictly during the training phase.
- Feature Extraction: Given synchronized multi-view observations, a frozen geometry foundation model (VGGT-Omega) estimates dense depth and camera parameters, while a vision foundation model (CLIP) extracts dense semantic features.
- Gaussian Field Unification: These multi-view geometric and semantic cues are mapped onto a shared grid and lifted into 3D space to initialize a 3D Gaussian field. Here, each Gaussian primitive jointly carries both geometric (position, scale, rotation, opacity) and semantic attributes.
- Rendering & Distillation: The optimized 3D Gaussian field is rendered back into 2D space to generate spatially aligned semantic, depth, and coverage (alpha) maps. These rendered targets are distilled into the WAM’s current-observation tokens via lightweight auxiliary prediction heads.
- Clean Inference: After training, the Gaussian field, teacher models, and auxiliary heads are completely removed. The final deployed policy uses the exact same architecture and forward path as the original WAM.
Datasets
The framework was evaluated across several simulation benchmarks and real-world tasks:
- LIBERO: A standard benchmark for language-conditioned robotic manipulation.
- LIBERO-Plus: An extension designed for zero-shot evaluation under severe distribution shifts (including camera viewpoint, lighting, background, visual noise, and layout changes).
- RoboTwin: A bimanual manipulation benchmark evaluated under both Clean and Random environments.
- Real-World Platform: A physical setup consisting of two UR7e robotic arms executing bimanual manipulation tasks under varying layouts and visual noise.
Key Insights and Ideas
- The Shared Spatial Carrier Advantage: Independent 2D distillation of geometry and semantics treats them as disconnected signals. Binding them to a shared 3D Gaussian field forces the model to consistently associate what an object is with where it is across multiple views.
- Privileged Training Scaffolding: Rich 3D representations can shape the internal latents of a standard 2D model without needing to be present at runtime. The Gaussian field acts as temporary scaffolding that is knocked away before deployment.
- Coverage-Guided Masking: By utilizing the rendered alpha channel as a coverage map, the distillation process restricts supervision exclusively to regions supported by valid geometry and non-negligible Gaussian contributions, preventing the model from learning from noisy or empty spatial zones.
Key Results
- Substantial Performance Boosts: On the challenging LIBERO-Plus benchmark, GaussianWAM improved the FastWAM baseline from 52.05% to 71.29% and the Cosmos Policy baseline from 71.52% to 77.30% in overall success rate.
- Unification vs. Direct Distillation: Direct 2D distillation of CLIP and VGGT features achieved a strong baseline of 69.37% on FastWAM. However, unifying these signals through the 3D Gaussian field pushed the performance further to 71.29%, validating the benefit of spatial organization.
- Real-World Validation: In physical bimanual robot experiments under distribution shifts, GaussianWAM improved the average success rate of FastWAM from 30.00% to 40.00% while maintaining identical inference speeds.
Limitations and Future Work
Not discussed in the paper.
Summary updated