Splat and Replace: 3D Reconstruction with Repetitive Elements

Nicolás Violante, Andreas Meuleman, Alban Gauthier, Frédo Durand, Thibault Groueix, George DrettakisSIGGRAPH (Conference Paper Track) 2025

TL;DR

This paper introduces "Splat and Replace," a novel method that significantly improves 3D scene reconstruction and novel view synthesis by leveraging repetitive elements within a scene. By segmenting, registering, and fusing these repeated instances into a shared representation, the method effectively fills in occluded areas and enhances low-quality geometry and appearance, outperforming existing state-of-the-art techniques.

What Problem Does This Paper Address?

Modern 3D reconstruction techniques like Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) have made impressive strides in generating novel views of scenes. However, these methods often struggle when training data is sparse or occluded, leading to low-quality renderings in unseen or hidden regions. This paper tackles this limitation by focusing on scenes that naturally contain repetitive elements, such as windows on a facade, chairs around a table, or pillars in a building. The core challenge is to effectively exploit the redundant information present in these repetitions to create a more complete and accurate 3D representation, even when individual instances are poorly captured. Existing methods have paid limited attention to this, and those that do often fail to account for inevitable variations in appearance (like lighting) between instances.

Methods

The "Splat and Replace" method operates in three main stages:

  1. 3D Instance Segmentation:
    • Starts with a base 3DGS reconstruction.
  2. Leverages pre-trained 2D segmentation models (SAM-HQ) guided by user clicks to identify instances of repetitive objects.
  3. Trains per-Gaussian features using contrastive learning to achieve 3D instance segmentation. This stage includes novel regularization techniques (opacity and scaling) and a post-processing step (space carving) to refine segmentation quality.
  4. Instance Registration:
    • The goal is to align all segmented instances into a common coordinate frame.
  5. Instead of relying on traditional 3D point-cloud matching (which struggles with noisy 3DGS data), this method exploits 3DGS's ability to render high-quality novel views.
  6. It generates additional viewpoints for each instance and uses robust 2D image matching (MASt3R) to find correspondences.
  7. These 2D-3D correspondences are then lifted to 3D using depth information from 3DGS, and a PnP-RANSAC solver is used to estimate the rigid transformation for each instance.
  8. Shared Representation and Optimization:
    • A "template" representation is created by taking the union of all registered instances.
  9. This shared representation is then optimized. Crucially, gradients from each instance flow back to this shared representation, allowing it to be completed and improved using information from all instances.
  10. Appearance variations (primarily due to illumination) are handled by decomposing Spherical Harmonics (SH) coefficients into a shared component and instance-specific offsets. This allows the method to account for differences in lighting while maintaining a common underlying geometry and appearance.

Datasets

The paper evaluates its method on a combination of synthetic and real-world datasets:

  • Synthetic Scenes: OFFICE, TEMPLE, CHESSBOARD, and CLASSROOM. These scenes were rendered using Blender and come with ground truth masks and depth maps for quantitative evaluation.
  • Real Scenes: HOUSE, MEETINGROOM, PILLARS, and FACADE. These scenes were captured and processed using COLMAP for calibration.
  • Additional Real Scenes: One scene from ScanNet++ and two from DL3DV, also featuring repetitive elements.

The paper notes that for real scenes, monocular depth is estimated using Depth-Anything-v2, and segmentation masks are generated using GroundingDINO and SAM-HQ.

Key Insights and Ideas

  • Repetitions as a Powerful Prior: The core insight is that repetitive elements in a scene are not just redundant but a valuable source of information that can significantly improve 3D reconstruction quality, especially in areas with poor coverage or occlusions.
  • Leveraging Novel View Synthesis for Registration: Traditional 3D registration methods struggle with the characteristics of 3DGS data. The paper cleverly sidesteps this by using the rendering capabilities of 3DGS to generate new views, enabling the use of robust 2D image matching techniques for more reliable 3D registration.
  • Shared Representation for Geometry and Appearance: The method effectively disentangles geometry and appearance. A shared representation captures the common geometry and base appearance, while instance-specific offsets handle variations like illumination, leading to more accurate and visually pleasing results.
  • User Interaction for Efficiency: While the method is automated, a minimal user interaction (a single click per instance) is required for segmentation, making the process practical and efficient.

Key Results

  • Quantitative Improvements: The method consistently outperforms baselines (Nerfbusters, Bayes' Rays, 3DGS*, and Ours) across various metrics (PSNR, SSIM, LPIPS, KID) on both synthetic and real scenes. For instance, on real scenes, it achieves an average improvement of 1.59 dB in PSNR over the second-best method, and 2.72 dB when considering only the masked regions of repetitive elements.
  • Qualitative Improvements: Visual results (shown in Figures 3, 4, 5, and 8) demonstrate significant enhancements in filling occluded areas, recovering fine details, and producing cleaner geometry and appearance for repetitive objects compared to baselines. The "Zoom-in" example (Fig. 5) highlights how a close-up view of a repetition can drastically improve the reconstruction of a distant, occluded instance.
  • Robust Registration: The proposed registration method achieves an average angular error of less than half a degree on synthetic scenes, significantly outperforming traditional methods like FPFH which struggle with 50-degree errors. Ablations show the importance of using novel views for matching and the effectiveness of ICP refinement.

Limitations and Future Work

  • Dependence on Repetitions: The method's effectiveness is directly tied to the presence and variability of repetitive elements in the scene. Scenes with few or highly dissimilar repetitions will benefit less.
  • Background Reconstruction: The method does not improve the reconstruction of the background, and artifacts may persist in these areas.
  • User Interaction: While minimal, user interaction is still required for instance identification.
  • Illumination Variability: Strong highlights and pronounced appearance differences due to illumination can still be challenging to model perfectly.
  • Computational Cost: The method requires additional training time for contrastive features and shared representation finetuning, though the authors note potential optimizations.
  • Future Work: The authors suggest exploring how the shared representation can reduce the memory footprint of 3DGS, and further investigating inverse rendering with the shared representation to disentangle material and illumination properties more effectively.

Summary updated

BibTeX
@inproceedings{violante2025splat,
  author = {Nicolás Violante and Andreas Meuleman and Alban Gauthier and Frédo Durand and Thibault Groueix and George Drettakis},
  title = {{Splat and Replace: 3D Reconstruction with Repetitive Elements}},
  booktitle = {SIGGRAPH (Conference Paper Track)},
  year = {2025},
}
Figure