GaussianWAM: Distilling Geometry and Semantics from 3D Gaussian Fields into World-Action Models

Zijian Zhang, Yuqing Jiang, Weitao Zhou, Minglei Li, Jinhao Zhang, Yao Mu, Xiaofan Li, Hao Zhao, Haibao YuarXiv 2026 · arXiv

TL;DR

GaussianWAM is a training-time enhancement framework that injects 3D geometric and semantic awareness into World-Action Models (WAMs) for robotic manipulation. By organizing heterogeneous teacher signals within a unified 3D Gaussian field during training, it significantly boosts policy robustness against visual and environmental shifts. Crucially, all 3D components are discarded before deployment, delivering these performance gains with zero additional inference overhead.

What Problem Does This Paper Address?

World-Action Models (WAMs) represent a powerful paradigm in robotics, combining future video prediction with action generation to learn implicit physical and temporal dynamics. However, because these models primarily operate within 2D video or image latent spaces, their representations are not explicitly constrained to preserve cross-view 3D geometric structures or localized object semantics.

While one could simply distill features from pretrained vision and geometry foundation models directly into the WAM, doing so treats geometry and semantics as disconnected 2D signals. Conversely, explicitly building 3D or 4D world models typically requires heavy architectural modifications, such as 3D encoders, decoders, or rendering modules, which drastically increase computational costs during deployment.

Comparison of WAM enhancement paradigms

Methods

The authors introduce GaussianWAM, a framework that leverages a 3D Gaussian field as a unified spatial carrier strictly during the training phase.

  1. Feature Extraction: Given synchronized multi-view observations, a frozen geometry foundation model (VGGT-Omega) estimates dense depth and camera parameters, while a vision foundation model (CLIP) extracts dense semantic features.
  2. Gaussian Field Unification: These multi-view geometric and semantic cues are mapped onto a shared grid and lifted into 3D space to initialize a 3D Gaussian field. Here, each Gaussian primitive jointly carries both geometric (position, scale, rotation, opacity) and semantic attributes.
  3. Rendering & Distillation: The optimized 3D Gaussian field is rendered back into 2D space to generate spatially aligned semantic, depth, and coverage (alpha) maps. These rendered targets are distilled into the WAM’s current-observation tokens via lightweight auxiliary prediction heads.
  4. Clean Inference: After training, the Gaussian field, teacher models, and auxiliary heads are completely removed. The final deployed policy uses the exact same architecture and forward path as the original WAM.
Overview of GaussianWAM framework

Datasets

The framework was evaluated across several simulation benchmarks and real-world tasks:

  • LIBERO: A standard benchmark for language-conditioned robotic manipulation.
  • LIBERO-Plus: An extension designed for zero-shot evaluation under severe distribution shifts (including camera viewpoint, lighting, background, visual noise, and layout changes).
  • RoboTwin: A bimanual manipulation benchmark evaluated under both Clean and Random environments.
  • Real-World Platform: A physical setup consisting of two UR7e robotic arms executing bimanual manipulation tasks under varying layouts and visual noise.

Key Insights and Ideas

  • The Shared Spatial Carrier Advantage: Independent 2D distillation of geometry and semantics treats them as disconnected signals. Binding them to a shared 3D Gaussian field forces the model to consistently associate what an object is with where it is across multiple views.
  • Privileged Training Scaffolding: Rich 3D representations can shape the internal latents of a standard 2D model without needing to be present at runtime. The Gaussian field acts as temporary scaffolding that is knocked away before deployment.
  • Coverage-Guided Masking: By utilizing the rendered alpha channel as a coverage map, the distillation process restricts supervision exclusively to regions supported by valid geometry and non-negligible Gaussian contributions, preventing the model from learning from noisy or empty spatial zones.

Key Results

  • Substantial Performance Boosts: On the challenging LIBERO-Plus benchmark, GaussianWAM improved the FastWAM baseline from 52.05% to 71.29% and the Cosmos Policy baseline from 71.52% to 77.30% in overall success rate.
  • Unification vs. Direct Distillation: Direct 2D distillation of CLIP and VGGT features achieved a strong baseline of 69.37% on FastWAM. However, unifying these signals through the 3D Gaussian field pushed the performance further to 71.29%, validating the benefit of spatial organization.
  • Real-World Validation: In physical bimanual robot experiments under distribution shifts, GaussianWAM improved the average success rate of FastWAM from 30.00% to 40.00% while maintaining identical inference speeds.
Main results on LIBERO and LIBERO-Plus

Limitations and Future Work

Not discussed in the paper.

Summary updated

Abstract

World-Action Models (WAMs) jointly learn future visual prediction and action generation, using video dynamics as a representation-learning signal for robotic manipulation. However, their video latents are primarily optimized for visual prediction and are not explicitly encouraged to preserve cross-view geometric structure or spatially localized, object-relevant semantics. We propose \textbf{GaussianWAM}, a training-time representation-enhancement framework that organizes geometric and semantic supervision through a 3D Gaussian field. Given synchronized multi-view observations, frozen geometry and vision foundation models provide depth, camera parameters, and dense semantic features. GaussianWAM binds these heterogeneous signals to shared Gaussian primitives and renders spatially aligned semantic, depth, and coverage targets, which are distilled into the current-observation representations of the WAM. All teacher models, Gaussian components, and auxiliary prediction heads are removed after training, leaving the original WAM inference path without additional modules or forward computation. On LIBERO-Plus, GaussianWAM improves FastWAM from 52.05\% to 71.29\% and Cosmos Policy from 71.52\% to 77.30\%. Direct CLIP and VGGT distillation already establishes a strong FastWAM baseline of 69.37\%, while Gaussian-field unification further improves it to 71.29\%, supporting the benefit of spatially organizing heterogeneous teacher signals. GaussianWAM also improves performance on standard LIBERO and shows positive transfer trends on RoboTwin and real-world manipulation. These results suggest that training-time Gaussian distillation provides a practical way to inject geometry- and semantics-related supervision into WAM representations without changing their deployment architecture.
BibTeX
@misc{zhang2026gaussianwam,
  author = {Zijian Zhang and Yuqing Jiang and Weitao Zhou and Minglei Li and Jinhao Zhang and Yao Mu and Xiaofan Li and Hao Zhao and Haibao Yu},
  title = {{GaussianWAM: Distilling Geometry and Semantics from 3D Gaussian Fields into World-Action Models}},
  year = {2026},
  eprint = {2608.24714},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.24714},
}
Figure