TL;DR
RigidGS is an end-to-end pipeline that enables clean, real-time object manipulation in 3D Gaussian Splatting (3DGS) by decoupling geometric learning from object identity. By refining 2D masks with depth data and freezing object labels before training, it eliminates the "label-geometry coupling" that typically causes messy artifacts and "ghosting" debris in interactive VR environments.
What Problem Does This Paper Address?
While 3D Gaussian Splatting has revolutionized real-time photorealistic rendering, it treats a scene as a monolithic "soup" of unstructured primitives. To make these scenes interactive—allowing a user to pick up a bear or move a truck—we need to know which Gaussians belong to which object.
Existing "object-aware" 3DGS methods usually try to learn these object identities (labels) and the scene geometry simultaneously. This leads to a major problem called label-geometry coupling: because the labels are being optimized alongside the shapes, boundary Gaussians often get "pulled" toward the wrong object, resulting in blurry edges or "debris" left behind when an object is moved. Furthermore, these methods rely on 2D foundation models like SAM2, which often fail on thin structures (over-inclusion) or objects with complex depth profiles (under-inclusion).
Methods
RigidGS introduces a three-step approach to decouple identity from geometry:
- Depth- and Temporally-Aware Mask Refinement: The authors improve raw SAM2 masks by using optical flow to warp masks between views and applying a depth-quantile filter (via Depth-Anything-V2). This fixes common errors where SAM2 might miss a truck bed or accidentally include background pixels between fern fronds.
- Fixed Object Identity: Instead of learning labels during training, RigidGS determines them entirely upstream. They use COLMAP projection voting on the refined masks to assign a permanent integer label to each 3D point. These labels are frozen; they never see a gradient during the main optimization.
- Geometry-Only Mask Supervision: During training, a 2D mask loss is used strictly to regularize the geometry (position, scale, opacity) of the Gaussians, forcing them to stay within the object's silhouette without the risk of changing their identity.
- Boundary Stabilization: To prevent different object "clouds" from interpenetrating, they add a lightweight stabilizer that penalizes variance along object interfaces and encourages Gaussians to form thin, non-overlapping discs at the boundaries.
Datasets
The authors evaluated RigidGS across several diverse 3DGS benchmarks:
- LLFF/NVOS: Scenes like fern, flower, and trex (noted for thin structures and complex silhouettes).
- Mip-NeRF 360: Garden and kitchen (cluttered indoor/outdoor scenes).
- Instruct-NeRF2NeRF: The bear scene.
- Tanks and Temples: Truck and train (multimodal-depth foregrounds).
- LERF: Figurines and teatime for multi-object segmentation.
Key Insights and Ideas
- The "Frozen" Advantage: The most significant "aha" moment is that not learning labels is actually better. By treating identity as a fixed attribute determined by multi-view consensus, the model avoids the optimization traps that cause background "bleeding."
- Depth as a Rescue Signal: SAM2's failures are often depth-related. By using depth quantiles, RigidGS can "rescue" parts of an object that a 2D-only model would truncate as being too far or too near.
- Mesh-Free Physics: Because the object separation is so clean, RigidGS doesn't need to extract a mesh for interaction. You can simply apply affine transformations (rotation, translation) to a subset of Gaussians and plug them directly into a standard game physics engine.
Key Results
RigidGS sets a new state-of-the-art for object-aware 3DGS:
- Segmentation Accuracy: On the LLFF/NVOS dataset, it improved Intersection-over-Union (IoU) by 7.7 points and Boundary IoU (bIoU) by 14.5 points over the previous best method, COB-GS.
- Clean Extraction: As shown in Figure 5, RigidGS achieves view-consistent background recovery after object removal, whereas previous methods like Gaussian Grouping leave "ghosting" residue.
- Real-Time VR: The system was deployed on a Meta Quest 2, maintaining ~33 FPS during static rendering and ~26 FPS during active rigid-body manipulation of complex scenes (e.g., the 4.03M Gaussian garden scene).
Limitations and Future Work
- Training Overhead: The boundary regularizer requires nearest-neighbor graph updates, which pushes offline training time to 2-3 hours per scene.
- VR Performance: While interactive, 26-33 FPS is still below the native 72Hz target for the Quest 2. The authors suggest future work could integrate foveated rendering or primitive pruning to bridge this gap.
- Rigid Only: As the name implies, the current framework focuses on rigid-body interactions. Extending this clean decoupling to articulated (e.g., a human arm) or non-rigid (e.g., cloth) dynamics is a promising next step.
Summary updated