VoteSplat: Hough Voting Gaussian Splatting for 3D Scene Understanding

Minchao Jiang, Shunyu Jia, Jiaming Gu, Xiaoyuan Lu, Guangming Zhu, Anqi Dong, Liang Zhang

Abstract

3D Gaussian Splatting (3DGS) has revolutionized real-time photorealistic rendering, yet it lacks inherent mechanisms for semantic scene understanding. We present VoteSplat, a framework that integrates the classic computer vision concept of Hough voting into the 3DGS pipeline. By training individual Gaussian primitives to "vote" for the 3D centers of the objects they belong to, VoteSplat enables precise instance segmentation and open-vocabulary object localization. This approach avoids the computational burden of high-dimensional feature embeddings, resulting in faster training and superior accuracy, especially in cluttered environments.

Introduction

While 3D Gaussian Splatting (3DGS) excels at rendering, it historically struggles to distinguish between individual objects within a scene. Standard 3DGS does not inherently know where one object ends and another begins, making tasks like instance segmentation difficult. Existing solutions often rely on attaching high-dimensional language features (like CLIP) to every Gaussian or using contrastive learning to separate object features. However, these methods face significant memory overhead and struggle with "spatial adjacency"—the difficulty of drawing clean boundaries between touching objects. VoteSplat addresses these issues by shifting the focus from object surfaces to their centers.

Methodology

VoteSplat transforms 3DGS into a spatial reasoning engine through a three-step pipeline:

  1. 2D Vote Construction: The framework utilizes the Segment Anything Model (SAM) to generate precise 2D masks for objects in training images. The centroids of these masks serve as ground-truth "2D votes."
  2. 3D Vote Splatting: Each Gaussian primitive is assigned a 3D offset vector ($\Delta p$). When added to the Gaussian's position, this vector points to a predicted 3D object center. During training, these 3D votes are projected (splatted) back into 2D space and supervised against the SAM-generated centroids.
  3. Regularization and Clustering: To resolve depth ambiguity from 2D projections, a custom depth distortion regularization loss forces voting points to concentrate tightly in 3D space. Post-training, HDBSCAN is used to cluster these 3D votes. Since all Gaussians in an object vote for the same central point, they form distinct clusters that are easily separated, even if the objects are physically touching.
The VoteSplat pipeline: (a) Rendered view with object centers, (b) Gaussian primitives voting for 3D centers, (c) Clustering of votes into distinct object instances, and (d) Final 3D scene understanding and hierarchical segmentation.

Experiments and Results

VoteSplat was evaluated on challenging benchmarks including LeRF, 3D-OVS, and GraspNet, setting new standards for efficiency and accuracy:

  • Accuracy: On the LeRF dataset, VoteSplat achieved a Mean IoU (Intersection over Union) of 50.10%, significantly outperforming competitors such as LangSplat (9.66%) and OpenGaussian (44.90%).
  • Speed: Training is remarkably efficient. For the "figurines" scene, VoteSplat trained in approximately 54 minutes, whereas Gaussian Grouping required ~150 minutes.
  • Robustness: The voting mechanism enforces global object consistency, allowing the model to accurately "fill in" occluded parts of objects by relying on the predicted central vote.

Conclusion

VoteSplat demonstrates that integrating Hough voting into 3D Gaussian Splatting provides a lightweight and effective solution for 3D scene understanding. By leveraging object centers rather than surface features, the model overcomes the boundary ambiguity problem common in cluttered scenes. Future work will address challenges related to large-scale objects that exceed the camera's field of view and complex concave geometries.

Summary updated

BibTeX
@inproceedings{jiang2025votesplat,
  author = {Minchao Jiang and Shunyu Jia and Jiaming Gu and Xiaoyuan Lu and Guangming Zhu and Anqi Dong and Liang Zhang},
  title = {{VoteSplat: Hough Voting Gaussian Splatting for 3D Scene Understanding}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
}
Figure