GauSSmart: Enhanced 3D Reconstruction through 2D Foundation Models and Geometric Filtering

Alexander Valverde, Brian Xu, Yuyin Zhou, Meng Xu, Hongyun WangarXiv 2025 · DOI · arXiv

Abstract

3D Gaussian Splatting (3DGS) has emerged as a leading technique for real-time neural rendering, yet it remains sensitive to noisy initializations and lacks high-level structural awareness. We present GauSSmart, a hybrid framework that integrates 2D foundation models and geometric filtering to enhance 3D reconstruction. By leveraging semantic priors from models like SAM and DINOv3, GauSSmart improves geometric fidelity and detail, particularly in scenes with sparse or noisy training data.

Introduction

While 3DGS and its variants have revolutionized the field of neural rendering, they often suffer from a "garbage in, garbage out" problem. These methods typically initialize using point clouds from Structure-from-Motion (SfM) tools like COLMAP, which are frequently riddled with noise, outliers, and uneven density. Standard optimization relies on low-level photometric losses (such as L1 or SSIM) that excel at pixel-matching but struggle to capture high-level semantic structures or fill gaps in sparse data. Consequently, reconstructions often exhibit "floaters," blurry textures, and poorly defined geometric boundaries.

Methodology

GauSSmart introduces a three-pronged approach to refine the reconstruction pipeline through "smart" priors and robust geometric cleaning.

1. Convex Outlier Removal

Traditional neighborhood-based filters often struggle with irregular point densities. GauSSmart employs a convex hull-based method that evaluates points independently. By removing points that fall outside the valid viewing cone, the system effectively eliminates the "halo" of noise common in SfM reconstructions, providing a cleaner foundation for the splatting process.

2. Segment-Aware Densification

To address uneven sampling, the framework utilizes the Segment Anything Model (SAM) to identify distinct objects and regions within a scene. Instead of uniform upsampling, GauSSmart calculates the area of these segments and adds new points based on a square-root relationship to that area. This ensures that visually prominent but sparsely sampled regions receive the necessary geometric budget.

Semantic segmentation of a scene using foundation models to identify distinct object boundaries.

3. Embedding-Aligned Training

Beyond simple pixel-matching, GauSSmart incorporates DINOv3 feature embeddings. A novel differentiable loss function based on cosine similarity is introduced to align rendered features with ground-truth DINO features. This forces the model to maintain semantic consistency across different viewpoints, leading to superior object coherence. Cosine similarity is preferred over L1 loss here because it focuses on the direction of the feature vector (the "what") rather than magnitude, isolating semantic correspondence from lighting variations.

4. Geometry-Aware Camera Clustering

To manage the computational cost of 2D foundation models, the authors propose a clustering method to select a compact subset of representative views for segmentation and feature extraction, ensuring efficiency without sacrificing reconstruction quality.

Experimental Results

The authors validated GauSSmart on standard benchmarks including DTU, Mip-NeRF 360, and Tanks and Temples.

  • Quantitative Performance: On the DTU dataset, GauSSmart achieved a mean PSNR of 36.30 dB, significantly outperforming 2DGS (34.52 dB) and 3DGS (35.76 dB).
  • Qualitative Improvements: Results show a marked reduction in "bubble-like" artifacts and floaters. Structural elements, such as walls and complex root systems (e.g., in the Treehill scene), were reconstructed with much higher fidelity than baseline methods.

Conclusion

GauSSmart demonstrates that 2D foundation models, trained on vast datasets, possess inherent knowledge of object structures that can compensate for deficiencies in 3D training data. While the integration of DINOv3 and SAM increases processing time by approximately 1.5x, the gains in geometric clarity and semantic consistency represent a significant step forward for neural reconstruction. Future work will focus on caching embeddings to reduce computational overhead and further optimizing the pipeline as foundation models continue to evolve.

Summary updated

BibTeX
@misc{valverde2025gaussmart,
  author = {Alexander Valverde and Brian Xu and Yuyin Zhou and Meng Xu and Hongyun Wang},
  title = {{GauSSmart: Enhanced 3D Reconstruction through 2D Foundation Models and Geometric Filtering}},
  year = {2025},
  eprint = {2510.14270},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.14270},
}
Figure