GaussianGrasper: 3D Language Gaussian Splatting for Open-Vocabulary Robotic Grasping

Yuhang Zheng, Xiangyu Chen, Yupeng Zheng, Songen Gu, Runyi Yang, Bu Jin, Pengfei Li, Chengliang Zhong, Zengmao Wang, Lina Liu, Chao Yang, Dawei Wang, Zhen Chen, Xiaoxiao Long, Meiqing Wang

Abstract

GaussianGrasper is a novel robotic manipulation system designed to bridge the gap between natural language instructions and physical grasping through the use of 3D Gaussian Splatting (3DGS). By distilling semantic features from large-scale foundation models into an explicit 3D representation, the system enables robots to localize and manipulate objects based on open-ended queries (e.g., "pick up the hamburger"). Experimental results demonstrate that GaussianGrasper achieves 180x faster inference and significantly higher localization accuracy than previous state-of-the-art methods like LERF, while maintaining the geometric precision necessary for stable grasping.

Introduction

For robots to function effectively in human-centric environments, they must possess two core capabilities: semantic understanding to interpret natural language and geometric precision to execute physical tasks. Traditional 2D-based vision models are computationally efficient but often struggle with spatial reasoning and occlusions. In contrast, 3D methods based on Neural Radiance Fields (NeRFs) provide rich semantic "heatmaps" but are computationally expensive, require extensive input data, and often produce "blurry" object boundaries that hinder precise grasping. Furthermore, NeRF-based representations are difficult to update quickly when the environment changes.

GaussianGrasper addresses these challenges by shifting the underlying representation from implicit neural fields to explicit 3D Gaussian Splatting. This transition allows for faster processing, sharper object boundaries, and the ability to update the scene in real-time as objects are moved.

Methodology

1. Efficient Feature Distillation (EFD)

To imbue the 3D scene with language understanding, GaussianGrasper distills features from foundation models—specifically CLIP for language and SAM (Segment Anything Model) for segmentation—into the 3D Gaussian primitives.

  • Contrastive Learning: The system utilizes SAM to identify object boundaries in 2D images. A contrastive loss function is then applied to ensure that all 3D points belonging to the same object share consistent language features, effectively sharpening the semantic boundaries.
  • Feature Compression: Storing high-dimensional CLIP vectors (e.g., 512 dimensions) for every Gaussian would result in prohibitive memory usage (exceeding 70GB). GaussianGrasper compresses these into low-dimensional latent features, which are decoded by a lightweight MLP during query time.

2. Normal-Guided Grasping

Localization is only the first step; the robot must also determine a stable grasp. GaussianGrasper renders surface normals—the direction a surface faces—directly from the 3D Gaussians. By applying force-closure theory, the system filters potential grasp candidates generated by models like AnyGrasp. It selects the pose that best aligns with the object's geometry, ensuring a more stable and reliable grip.

3. Rapid Scene Updating

Because 3DGS is an explicit collection of primitives, the representation is inherently editable. When an object is moved, the system can transform the specific Gaussians associated with that object. The scene can then be fine-tuned with as few as five new camera views, allowing the robot to maintain an accurate map of its environment without a full reconstruction.

Experimental Evaluation

The system was evaluated in 120 real-world trials across 10 distinct desktop scenes containing 44 unique objects. Performance was compared against LSeg (a 2D semantic segmentation baseline) and LERF (the previous 3D language-field standard).

Key Results

  • Inference Speed: GaussianGrasper performs language queries in approximately 0.22 seconds, representing a 180x speedup over LERF (40.27s).
  • Localization Accuracy: The system achieved a localization accuracy of 87.5%, significantly outperforming LERF’s 65.1%.
  • Grasping Success: The introduction of the normal-guided module improved the grasping success rate from 78.3% to 85.0%.
  • Data Efficiency: The system requires only 16 initial views to construct a scene and just 5 views for a rapid update after an object is moved.

Discussion and Limitations

While GaussianGrasper represents a significant leap in open-vocabulary grasping, certain limitations remain. The system currently assumes the scene is static during the update process, meaning it cannot yet account for dynamic changes that occur while the robot is actively moving. Additionally, like many vision-based 3D reconstruction techniques, it struggles with transparent or highly reflective objects (e.g., glassware) due to the limitations of depth sensors and the way light interacts with such surfaces.

Future work will focus on extending the system to handle "long-horizon" tasks, where the robot must reason through a sequence of complex language instructions to complete a multi-step objective.

Conclusion

GaussianGrasper demonstrates that explicit 3D representations like Gaussian Splatting are superior to implicit fields for real-time robotic applications. By combining semantic distillation with geometry-aware grasping filters, the system provides a robust framework for robots to interact with the world using natural language.

Summary updated

BibTeX
@article{zheng2024gaussiangrasper,
  author = {Yuhang Zheng and Xiangyu Chen and Yupeng Zheng and Songen Gu and Runyi Yang and Bu Jin and Pengfei Li and Chengliang Zhong and Zengmao Wang and Lina Liu and Chao Yang and Dawei Wang and Zhen Chen and Xiaoxiao Long and Meiqing Wang},
  title = {{GaussianGrasper: 3D Language Gaussian Splatting for Open-Vocabulary Robotic Grasping}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2024},
  doi = {10.1109/lra.2024.3432348},
}
Figure