VoteSplat: Hough Voting Gaussian Splatting for 3D Scene Understanding

Minchao Jiang, Shunyu Jia, Jiaming Gu, Xiaoyuan Lu, Guangming Zhu, Anqi Dong, Liang Zhang

TL;DR

This paper introduces VoteSplat, a novel framework that enhances 3D Gaussian Splatting (3DGS) for robust 3D scene understanding. By integrating Hough voting with 3DGS, VoteSplat enables efficient, open-vocabulary instance segmentation and localization without relying on high-dimensional features, significantly improving training efficiency and accuracy.

What Problem Does This Paper Address?

3D Gaussian Splatting (3DGS) has revolutionized real-time novel view synthesis, but its application to deeper scene understanding has been limited. Existing methods either embed high-dimensional semantic features (leading to ambiguity and high training costs) or use complex, multi-stage pipelines with contrastive learning for instance differentiation. A key challenge is accurately segmenting and localizing objects in 3D, especially when object centers are not directly represented by surface points, making traditional point cloud clustering difficult. VoteSplat aims to bridge this gap by enabling efficient and accurate 3D scene understanding directly within the streamlined 3DGS framework.

Methods

VoteSplat builds upon 3DGS by introducing a Hough voting mechanism for instance segmentation and localization. The core components are:

  • Instance Segmentation with SAM: The Segment Anything Model (SAM) is used to automatically generate high-quality instance masks from input images.
  • 2D Vote Construction: For each segmented mask, the centroid is computed, serving as a 2D ground-truth vote. These 2D votes are crucial for supervising the learning of 3D votes.
  • 3D Vote Construction: Each 3D Gaussian primitive in the 3DGS model is augmented with a learnable 3D spatial offset vector. These offsets allow the primitives to "vote" towards the instance center.
  • 3D-2D Votes Association Learning: This novel approach links the 2D image votes with the 3D votes generated by the Gaussians. This association is optimized using a voting loss.
  • Depth Regularization: To mitigate depth information loss during projection and improve vote aggregation along the depth axis, a depth distortion regularization term is introduced. This ensures that votes are more tightly clustered around the true instance center in 3D space.
  • Semantic Association: For open-vocabulary understanding, VoteSplat maps 2D image semantics (derived from CLIP) to 3D Gaussians via their instance IDs. This avoids the need for high-dimensional CLIP features directly embedded in Gaussians, reducing training costs.
  • Clustering and Instance Dictionary: HDB-SCAN is used to cluster the 3D votes, creating an instance dictionary that maps instance IDs to Gaussian primitives. This enables efficient instance retrieval and manipulation.

Datasets

The paper evaluates VoteSplat on the following datasets:

  • 3D-OVS: Used for experiments on open-vocabulary object selection and ablation studies.
  • LeRF-OVS: Used for open-vocabulary object selection and click-based 3D object selection/editing.
  • GraspNet: Used for evaluating instance segmentation in complex scenes with overlapping and contained instances.

Key Insights and Ideas

  • Hough Voting for 3DGS: The integration of Hough voting into the 3DGS pipeline is a key innovation, allowing for point-level instance segmentation without complex multi-stage processing or high-dimensional feature embeddings.
  • Leveraging 2D Semantics for 3D: Using SAM to generate precise 2D masks and their centroids as ground-truth votes is an effective way to guide the learning of 3D votes, overcoming the limitations of sparse 3D point clouds.
  • Depth Regularization is Crucial: The depth distortion regularization term is vital for ensuring accurate 3D vote aggregation, especially in scenarios where projection can lead to depth ambiguities.
  • Efficient Open-Vocabulary Understanding: By mapping instance IDs to CLIP features rather than directly embedding high-dimensional features into Gaussians, VoteSplat achieves efficient and accurate open-vocabulary scene understanding.

Key Results

  • Superior Semantic Segmentation: VoteSplat outperforms existing methods like LangSplat and OpenGaussian in semantic segmentation tasks, achieving higher mIoU and mAcc scores on the LeRF dataset.
  • Faster Training: VoteSplat demonstrates significantly shorter training times compared to methods like OpenGaussian and Gaussian Grouping, attributed to its efficient voting mechanism and avoidance of complex contrastive learning.
  • Robust Instance Segmentation: The method successfully handles complex scenes with overlapping and contained instances, as shown on the GraspNet dataset.
  • Effective Open-Vocabulary Localization: VoteSplat accurately identifies and localizes objects based on text queries, even for fine-grained details and occluded objects.
  • Click-based Editing: The framework enables intuitive click-based 3D object selection and scene editing.

Limitations and Future Work

  • Projection Challenges: The voting mechanism can encounter challenges when instances have projected sizes that significantly exceed the field of view (FoV), leading to inaccuracies in 2D votes.
  • Concave Containers: VoteSplat may struggle with instances enclosed in concave containers, where spatial proximity makes precise separation difficult.
  • Future Work: The authors suggest that these limitations point towards future research directions, particularly in improving robustness to extreme aspect ratios and complex containment scenarios.

Summary updated

Abstract

3D Gaussian Splatting (3DGS) has become horsepower in high-quality, real-time rendering for novel view synthesis of 3D scenes. However, existing methods focus primarily on geometric and appearance modeling, lacking deeper scene understanding while also incurring high training costs that complicate the originally streamlined differentiable rendering pipeline. To this end, we propose VoteSplat, a novel 3D scene understanding framework that integrates Hough voting with 3DGS. Specifically, Segment Anything Model (SAM) is utilized for instance segmentation, extracting objects, and generating 2D vote maps. We then embed spatial offset vectors into Gaussian primitives. These offsets construct 3D spatial votes by associating them with 2D image votes, while depth distortion constraints refine localization along the depth axis. For open-vocabulary object localization, VoteSplat maps 2D image semantics to 3D point clouds via voting points, reducing training costs associated with high-dimensional CLIP features while preserving semantic unambiguity. Extensive experiments demonstrate effectiveness of VoteSplat in open-vocabulary 3D instance localization, 3D point cloud understanding, click-based 3D object localization, hierarchical segmentation, and ablation studies. Our code is available at https://sy-ja.github.io/votesplat/
BibTeX
@inproceedings{jiang2025votesplat,
  author = {Minchao Jiang and Shunyu Jia and Jiaming Gu and Xiaoyuan Lu and Guangming Zhu and Anqi Dong and Liang Zhang},
  title = {{VoteSplat: Hough Voting Gaussian Splatting for 3D Scene Understanding}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
}
Figure