Abstract
3D Gaussian Splatting (3DGS) has revolutionized real-time photorealistic rendering, yet it lacks inherent mechanisms for semantic scene understanding. We present VoteSplat, a framework that integrates the classic computer vision concept of Hough voting into the 3DGS pipeline. By training individual Gaussian primitives to "vote" for the 3D centers of the objects they belong to, VoteSplat enables precise instance segmentation and open-vocabulary object localization. This approach avoids the computational burden of high-dimensional feature embeddings, resulting in faster training and superior accuracy, especially in cluttered environments.
Introduction
While 3D Gaussian Splatting (3DGS) excels at rendering, it historically struggles to distinguish between individual objects within a scene. Standard 3DGS does not inherently know where one object ends and another begins, making tasks like instance segmentation difficult. Existing solutions often rely on attaching high-dimensional language features (like CLIP) to every Gaussian or using contrastive learning to separate object features. However, these methods face significant memory overhead and struggle with "spatial adjacency"—the difficulty of drawing clean boundaries between touching objects. VoteSplat addresses these issues by shifting the focus from object surfaces to their centers.
Methodology
VoteSplat transforms 3DGS into a spatial reasoning engine through a three-step pipeline:
- 2D Vote Construction: The framework utilizes the Segment Anything Model (SAM) to generate precise 2D masks for objects in training images. The centroids of these masks serve as ground-truth "2D votes."
- 3D Vote Splatting: Each Gaussian primitive is assigned a 3D offset vector ($\Delta p$). When added to the Gaussian's position, this vector points to a predicted 3D object center. During training, these 3D votes are projected (splatted) back into 2D space and supervised against the SAM-generated centroids.
- Regularization and Clustering: To resolve depth ambiguity from 2D projections, a custom depth distortion regularization loss forces voting points to concentrate tightly in 3D space. Post-training, HDBSCAN is used to cluster these 3D votes. Since all Gaussians in an object vote for the same central point, they form distinct clusters that are easily separated, even if the objects are physically touching.
Experiments and Results
VoteSplat was evaluated on challenging benchmarks including LeRF, 3D-OVS, and GraspNet, setting new standards for efficiency and accuracy:
- Accuracy: On the LeRF dataset, VoteSplat achieved a Mean IoU (Intersection over Union) of 50.10%, significantly outperforming competitors such as LangSplat (9.66%) and OpenGaussian (44.90%).
- Speed: Training is remarkably efficient. For the "figurines" scene, VoteSplat trained in approximately 54 minutes, whereas Gaussian Grouping required ~150 minutes.
- Robustness: The voting mechanism enforces global object consistency, allowing the model to accurately "fill in" occluded parts of objects by relying on the predicted central vote.
Conclusion
VoteSplat demonstrates that integrating Hough voting into 3D Gaussian Splatting provides a lightweight and effective solution for 3D scene understanding. By leveraging object centers rather than surface features, the model overcomes the boundary ambiguity problem common in cluttered scenes. Future work will address challenges related to large-scale objects that exceed the camera's field of view and complex concave geometries.
Summary updated