Abstract
Robotic manipulation in dynamic household environments requires a simultaneous understanding of geometry, semantics, and motion. Current representations like Neural Radiance Fields (NeRFs) and 3D Gaussian Splatting (3DGS) often struggle with real-time updates or precise surface representation. We present MSGField, a unified scene representation based on 2D Gaussian Splatting. MSGField integrates an object-centric semantic field for open-vocabulary tasks and a compact motion field using learnable bases for efficient tracking. Our approach enables high-fidelity manipulation of rigid and non-rigid objects, including thin items like cable ties, using only two RGB camera views. Experimental results demonstrate a 79.2% success rate in static scenes and significant improvements in dynamic tracking speed and accuracy.
Introduction
For robots to effectively perform household tasks, they must perceive their environment with high fidelity across three dimensions: geometry (where objects are), semantics (what objects are), and motion (how objects move). Existing methods typically prioritize one aspect at the expense of others. For instance, Neural Radiance Fields (NeRFs) offer excellent semantic integration but are computationally intensive, hindering real-time performance. Conversely, 3D Gaussian Splatting (3DGS) provides speed but often results in "cloudy" volumetric representations that lack the surface precision necessary for delicate grasping tasks.
MSGField addresses these challenges by leveraging 2D Gaussian Splatting (2DGS) as a foundation. Unlike 3D primitives, 2DGS utilizes flat primitives that more accurately represent physical surfaces. This choice is critical for robotic interaction, where the difference between a successful grasp and failure often depends on millimeter-level precision.
Methodology
MSGField introduces a unified pipeline that decomposes the scene into semantic and motion components while maintaining a consistent geometric backbone.
Object-Centric Semantic Field
To avoid the memory overhead of per-point high-dimensional features, MSGField employs an object-centric distillation process. Using YOLO-World and Segment Anything Model (SAM), the system generates object masks from 2D views. These labels are then projected onto the 2D Gaussians. This approach ensures that all primitives belonging to a single object share a discrete, consistent label, facilitating open-vocabulary queries (e.g., "grasp the green dinosaur") without the "semantic blending" artifacts common in NeRF-based methods.
Compact Motion Field
Motion is represented as a linear combination of a small set of learnable motion bases. This formulation significantly reduces the degrees of freedom in the optimization problem, acting as a regularizer that prevents the scene representation from deforming chaotically. By leveraging differentiable rendering, MSGField can optimize the motion of rigid bodies in approximately 5 seconds using only two RGB camera views.
Optimization and Alignment
To handle large displacements between frames, the authors incorporate a Dice Loss for global alignment. This allows the Gaussian "cloud" to accurately jump to a new global position before fine-tuning local details. Furthermore, the use of 2DGS includes a depth distortion loss that prevents Gaussians from manifesting in empty space, ensuring the robot interacts with actual surfaces.
Experimental Results
The system was evaluated on a real-world dataset featuring 30 diverse object categories, ranging from daily items to flexible toys and extremely thin objects like a 5mm cable tie.
Static and Dynamic Performance
In static scene benchmarks, MSGField achieved a 79.2% success rate, significantly outperforming the NeRF-based F3RM baseline (49.2%). The system demonstrated remarkable precision, achieving a 90% success rate for target-specific grasping of thin objects that typically challenge standard RGB-D sensors.
In dynamic scenarios, MSGField successfully tracked non-rigid deformations—such as a toy dinosaur changing posture—with a 43.3% success rate. While non-rigid tracking remains a complex challenge, this performance represents a substantial improvement over existing baselines that often fail entirely in such settings.
Efficiency
The semantic field can be initialized in 30 seconds, and motion updates for rigid bodies occur in near real-time (5 seconds). This efficiency makes MSGField a viable candidate for deployment in interactive robotic systems.
Conclusion
MSGField represents a significant step toward unified scene representations for robotics. By integrating 2D Gaussian Splatting with efficient semantic distillation and motion basis optimization, it provides the speed and precision required for complex manipulation tasks. Future work will address current limitations, such as the "unseen side" problem—where the representation fails when an object reveals previously unobserved geometry—and further optimization for mobile robotic platforms with dynamic camera viewpoints.
Summary updated