Abstract
FHGS (Feature-Homogenized Gaussian Splatting) addresses the inherent conflict between the view-dependent (anisotropic) rendering of 3D Gaussian Splatting and the view-independent (isotropic) nature of semantic features. By treating high-dimensional semantic features as non-differentiable and employing a physics-inspired optimization approach, FHGS achieves highly accurate, multi-view consistent 3D semantic fields. The framework maintains real-time rendering speeds and drastically reduces training time compared to existing state-of-the-art methods.
Introduction: The Appearance vs. Semantics Conflict
3D Gaussian Splatting (3DGS) has revolutionized novel view synthesis by offering explicit scene representation and real-time rendering. Integrating high-level semantic understanding from foundation models like SAM or CLIP into these scenes is a natural progression. However, a fundamental contradiction exists between appearance and semantics.
To model complex lighting and reflections, 3DGS relies on anisotropic Gaussians where color changes based on the viewing angle. Conversely, semantic features must be isotropic—a chair's identity should not change based on the observer's position. Existing methods often attempt to solve this by making semantic features differentiable, which frequently distorts high-dimensional embeddings, leading to noise, poor cross-view consistency, and excessive computational overhead.
Methodology
FHGS introduces a novel fusion strategy to embed 2D foundation model features into 3D space without compromising efficiency or semantic integrity.
System Overview
The pipeline begins with sparse point cloud initialization via SFM, followed by a pre-training phase that incorporates feature ground truth extracted from 2D models.
Non-Differentiable Feature Driving (NDFD)
Instead of updating semantic feature vectors via backpropagation, FHGS freezes the features extracted from 2D models. During training, loss gradients flow through the rasterization weights to update the geometry and opacity of the Gaussians. This ensures that the rich relationships learned by models like SAM are preserved while the physical structure of the scene is optimized to match the semantic ground truth.
Physics-Inspired Dual-Drive Mechanism
The optimization of Gaussian placement is guided by an analogy to electric fields:
- External Potential Field Constraint: Ground-truth 2D features act as an electric field. Gaussians are treated as "charges" pulled toward regions where their frozen features match the ground truth.
- Internal Clustering Driving: Along a rendering ray, Gaussians with similar features attract each other, while dissimilar ones repel. This acts as a regularization term that suppresses noise and ensures local semantic coherence in $O(N)$ time.
Key Results and Evaluation
FHGS was evaluated on standard benchmarks including DTU, Mip-NeRF 360, and Tanks and Temples. The method demonstrates significant improvements across several metrics:
- Efficiency: Training time is reduced by approximately 15x compared to the Feature3DGS baseline (e.g., 5 minutes vs. 82 minutes on DTU). It maintains real-time rendering speeds of $\geq$ 60 FPS.
- Geometric Accuracy: Semantic guidance improves underlying 3D geometry. On the DTU dataset, FHGS achieved a Chamfer Distance of 0.75, significantly better than standard 3DGS (1.96).
- Memory Footprint: FHGS produces more compact models. For DTU scan 24, the model size was 183 MB compared to 745 MB for Feature3DGS.
- Qualitative Quality: Visual results show sharper semantic boundaries and a marked reduction in background clutter compared to differentiable feature methods.
Conclusion
FHGS demonstrates that geometry can be effectively driven by semantics. By refusing to alter high-dimensional feature vectors, the framework preserves semantic integrity while achieving superior speed and accuracy. Future work will focus on adaptive parameter learning to reduce manual tuning and further optimizing memory buffers for large-scale environment scaling.
Summary updated