FAST-Splat: Fast, Ambiguity-Free Semantics Transfer in Gaussian Splatting

Ola Shorinwa, Jiankai Sun, Mac SchwagerarXiv 2024 · DOI · arXiv

Introduction

3D Gaussian Splatting (GSplat) has revolutionized real-time scene reconstruction with its high-speed rendering. However, integrating semantic layers—the ability to identify specific objects within a scene—has traditionally introduced significant bottlenecks in speed, memory consumption, and semantic clarity. Existing open-vocabulary systems often suffer from language ambiguity, where a query for "tea" might highlight a coffee machine without clarifying the distinction. FAST-Splat addresses these issues through a high-speed, memory-efficient, and "neural-free" semantic framework.

Methodology: Neural-Free Semantics

FAST-Splat moves away from the implicit neural fields used in prior work, opting for an explicit representation. Each individual Gaussian "splat" is augmented with a 3D semantic code ($\beta$). This architecture allows for single-phase training, where geometry, color, and semantics are optimized simultaneously, rather than in separate, time-consuming stages.

Key components include:

  • Semantic Hash-Table: The system maintains a dictionary of identified objects. An affine transformation maps the Gaussian's 3D code to a probability distribution over this dictionary.
  • Open-Vocabulary Grounding: At runtime, pre-trained vision-language models (like CLIP) compute embeddings for user queries. The system then calculates a relevancy score to find the best match in its internal dictionary, providing an explicit label rather than a simple heatmap.
FAST-Splat resolving semantic ambiguity in natural language queries.

Experimental Evaluation

The researchers evaluated FAST-Splat on standard benchmarks (3D-OVS, LERF, MipNeRF360) and introduced new real-world datasets, R-Kitchen and R-Library, to test performance in complex, uncurated environments.

Key Results

FAST-Splat significantly outperforms state-of-the-art baselines such as LangSplat and Feature-3DGS across all primary efficiency metrics:

  • Rendering Speed: Achieves 115–128 FPS, representing an 18x to 51x speedup over competitors.
  • Memory Efficiency: Requires only ~1.0–1.2 GB of GPU VRAM, a 6x reduction compared to the 7–8 GB required by previous methods.
  • Training Time: Training is 6x to 8x faster, completing complex scenes in roughly 14 minutes.
Performance comparison showing FAST-Splat's superior speed and memory efficiency.

Discussion and Limitations

The core insight of FAST-Splat is that explicit semantic representation mirrors the efficiency of explicit geometry in Gaussian Splatting. By attaching semantic codes directly to primitives, the system bypasses expensive neural network queries during rendering. Furthermore, by providing specific labels (e.g., identifying an "apple" when queried for "fruit"), FAST-Splat mitigates the "semantic drift" common in heat-map-based methods.

Despite these gains, the system remains dependent on the quality of the underlying 2D foundation models (CLIP or SAM-2). Future work will focus on uncertainty estimation for safety-critical applications and extending the framework to dynamic, moving scenes.

Conclusion

FAST-Splat demonstrates that high-fidelity 3D semantic understanding does not require heavy neural overhead. By leveraging explicit Gaussian primitives, it provides a fast, memory-efficient, and unambiguous solution for real-time scene analysis.

Summary updated

Abstract

We present FAST-Splat for fast, ambiguity-free semantic Gaussian Splatting, which seeks to address the main limitations of existing semantic Gaussian Splatting methods, namely: slow training and rendering speeds; high memory usage; and ambiguous semantic object localization. We take a bottom-up approach in deriving FAST-Splat, dismantling the limitations of closed-set semantic distillation to enable open-set (open-vocabulary) semantic distillation. Ultimately, this key approach enables FAST-Splat to provide precise semantic object localization results, even when prompted with ambiguous user-provided natural-language queries. Further, by exploiting the explicit form of the Gaussian Splatting scene representation to the fullest extent, FAST-Splat retains the remarkable training and rendering speeds of Gaussian Splatting. Precisely, while existing semantic Gaussian Splatting methods distill semantics into a separate neural field or utilize neural models for dimensionality reduction, FAST-Splat directly augments each Gaussian with specific semantic codes, preserving the training, rendering, and memory-usage advantages of Gaussian Splatting over neural field methods. These Gaussian-specific semantic codes, together with a hash-table, enable semantic similarity to be measured with open-vocabulary user prompts and further enable FAST-Splat to respond with unambiguous semantic object labels and $3$D masks, unlike prior methods. In experiments, we demonstrate that FAST-Splat is 6x to 8x faster to train, achieves between 18x to 51x faster rendering speeds, and requires about 6x smaller GPU memory, compared to the best-competing semantic Gaussian Splatting methods. Further, FAST-Splat achieves relatively similar or better semantic segmentation performance compared to existing methods. After the review period, we will provide links to the project website and the codebase.
BibTeX
@misc{shorinwa2024fast,
  author = {Ola Shorinwa and Jiankai Sun and Mac Schwager},
  title = {{FAST-Splat: Fast, Ambiguity-Free Semantics Transfer in Gaussian Splatting}},
  year = {2024},
  eprint = {2411.13753},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.13753},
}
Figure