FHGS: Feature-Homogenized Gaussian Splatting

Q.g. Duan, Benyun Zhao, Mingqiao Han, Yijun Huang, Ben M. Chen

Abstract

FHGS (Feature-Homogenized Gaussian Splatting) addresses the inherent conflict between the view-dependent (anisotropic) rendering of 3D Gaussian Splatting and the view-independent (isotropic) nature of semantic features. By treating high-dimensional semantic features as non-differentiable and employing a physics-inspired optimization approach, FHGS achieves highly accurate, multi-view consistent 3D semantic fields. The framework maintains real-time rendering speeds and drastically reduces training time compared to existing state-of-the-art methods.

Introduction: The Appearance vs. Semantics Conflict

3D Gaussian Splatting (3DGS) has revolutionized novel view synthesis by offering explicit scene representation and real-time rendering. Integrating high-level semantic understanding from foundation models like SAM or CLIP into these scenes is a natural progression. However, a fundamental contradiction exists between appearance and semantics.

To model complex lighting and reflections, 3DGS relies on anisotropic Gaussians where color changes based on the viewing angle. Conversely, semantic features must be isotropic—a chair's identity should not change based on the observer's position. Existing methods often attempt to solve this by making semantic features differentiable, which frequently distorts high-dimensional embeddings, leading to noise, poor cross-view consistency, and excessive computational overhead.

Methodology

FHGS introduces a novel fusion strategy to embed 2D foundation model features into 3D space without compromising efficiency or semantic integrity.

System Overview

The pipeline begins with sparse point cloud initialization via SFM, followed by a pre-training phase that incorporates feature ground truth extracted from 2D models.

Non-Differentiable Feature Driving (NDFD)

Instead of updating semantic feature vectors via backpropagation, FHGS freezes the features extracted from 2D models. During training, loss gradients flow through the rasterization weights to update the geometry and opacity of the Gaussians. This ensures that the rich relationships learned by models like SAM are preserved while the physical structure of the scene is optimized to match the semantic ground truth.

Physics-Inspired Dual-Drive Mechanism

The optimization of Gaussian placement is guided by an analogy to electric fields:

  1. External Potential Field Constraint: Ground-truth 2D features act as an electric field. Gaussians are treated as "charges" pulled toward regions where their frozen features match the ground truth.
  2. Internal Clustering Driving: Along a rendering ray, Gaussians with similar features attract each other, while dissimilar ones repel. This acts as a regularization term that suppresses noise and ensures local semantic coherence in $O(N)$ time.

Key Results and Evaluation

FHGS was evaluated on standard benchmarks including DTU, Mip-NeRF 360, and Tanks and Temples. The method demonstrates significant improvements across several metrics:

  • Efficiency: Training time is reduced by approximately 15x compared to the Feature3DGS baseline (e.g., 5 minutes vs. 82 minutes on DTU). It maintains real-time rendering speeds of $\geq$ 60 FPS.
  • Geometric Accuracy: Semantic guidance improves underlying 3D geometry. On the DTU dataset, FHGS achieved a Chamfer Distance of 0.75, significantly better than standard 3DGS (1.96).
  • Memory Footprint: FHGS produces more compact models. For DTU scan 24, the model size was 183 MB compared to 745 MB for Feature3DGS.
  • Qualitative Quality: Visual results show sharper semantic boundaries and a marked reduction in background clutter compared to differentiable feature methods.

Conclusion

FHGS demonstrates that geometry can be effectively driven by semantics. By refusing to alter high-dimensional feature vectors, the framework preserves semantic integrity while achieving superior speed and accuracy. Future work will focus on adaptive parameter learning to reduce manual tuning and further optimizing memory buffers for large-scale environment scaling.

Summary updated

Abstract

Scene understanding based on 3D Gaussian Splatting (3DGS) has recently achieved notable advances. Although 3DGS related methods have efficient rendering capabilities, they fail to address the inherent contradiction between the anisotropic color representation of gaussian primitives and the isotropic requirements of semantic features, leading to insufficient cross-view feature consistency. To overcome the limitation, we proposes FHGS (Feature-Homogenized Gaussian Splatting), a novel 3D feature distillation framework inspired by physical models, which freezes and distills 2D pre-trained features into 3D representations while preserving the real-time rendering efficiency of 3DGS. Specifically, our FHGS introduces the following innovations: Firstly, a universal feature fusion architecture is proposed, enabling robust embedding of large-scale pre-trained models' semantic features (e.g., SAM, CLIP) into sparse 3D structures. Secondly, a non-differentiable feature fusion mechanism is introduced, which enables semantic features to exhibit viewpoint independent isotropic distributions. This fundamentally balances the anisotropic rendering of gaussian primitives and the isotropic expression of features; Thirdly, a dual-driven optimization strategy inspired by electric potential fields is proposed, which combines external supervision from semantic feature fields with internal primitive clustering guidance. This mechanism enables synergistic optimization of global semantic alignment and local structural consistency. Extensive comparison experiments with other state-of-the-art methods on benchmark datasets demonstrate that our FHGS exhibits superior reconstruction performance in feature fusion, noise suppression, and geometric precision, while maintaining a significantly lower training time. This work establishes a novel Gaussian Splatting data structure, offering practical advancements for real-time semantic mapping, 3D stylization, and Vision-Language Navigation (VLN). Our code and additional results are available on our project page:https://fhgs.cuastro.org/.
BibTeX
@inproceedings{duan2025fhgs,
  author = {Q.G. Duan and Benyun ZHAO and Mingqiao Han and Yijun Huang and Ben M. Chen},
  title = {{FHGS: Feature-Homogenized Gaussian Splatting}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=BEYzDKMEhX},
}
Figure