Introduction
3D Gaussian Splatting (3DGS) has significantly advanced real-time novel view synthesis; however, it remains highly dependent on dense input data. In indoor "inside-out" scenarios where camera overlap is naturally low, 3DGS often fails when provided with sparse inputs (e.g., as few as 12 images). These failures typically manifest as sparse point cloud initialization and insufficient supervision, leading to "floaters," blurry artifacts, and semantic ambiguity in unobserved regions.
SPC-GS addresses these challenges by integrating generative video models for robust initialization and leveraging Segment Anything Model 2 (SAM2) to enforce semantic consistency across views.
Methodology
Scene-layout-based Gaussian Initialization (SGI)
Standard Structure-from-Motion (SfM) often fails to produce a dense point cloud from sparse indoor snapshots. SPC-GS introduces SGI, which utilizes a Video Generation Model (VGM), specifically MotionCtrl, to "hallucinate" additional perspectives. By generating short video clips (pan, tilt, zoom) for each training image, the system creates a much denser initial 3D point cloud. This process is further refined using an Outlier Gaussian Primitive Removal (OGR) strategy to eliminate geometric noise.
Semantic-Prompt Consistency (SPC) Regularization
To maintain semantic coherence in "open-world" scenarios, SPC-GS employs SAM2 as a consistency teacher. The method uses Iterative Stochastic Prompting (ISP) to track object masks between known views and rendered pseudo-views from unobserved angles. By applying 2D and 3D semantic constraints—including region boundary erosion to prevent feature "bleeding"—the model ensures that objects like furniture and architectural elements are correctly identified from any viewpoint.
Experimental Results
The researchers evaluated SPC-GS on the Replica and ScanNet datasets using a strict 12-view protocol.
Quantitative Performance
SPC-GS sets a new state-of-the-art for sparse-input indoor scenes. On the Replica dataset, it achieved a PSNR of 22.011 dB, a significant improvement over previous methods. Furthermore, the model demonstrated a 7.3% increase in mean Intersection over Union (mIoU) for open-world segmentation compared to Gau-Grouping, highlighting its superior semantic understanding.
Conclusion
SPC-GS effectively bridges the gap between sparse data and high-quality 3D reconstruction. By treating generative AI as a geometric prior and SAM2 as a semantic bridge, the framework achieves superior visual fidelity and robust object recognition. While currently limited to static scenes and dependent on the physical plausibility of video generation models, SPC-GS represents a major step toward practical, sparse-view indoor scene understanding.
Summary updated