Complete Gaussian Splats from a Single Image with Denoising Diffusion Models

TL;DR

This paper introduces a novel generative approach to reconstruct complete 3D scenes from a single RGB image using a latent diffusion model over 3D Gaussian Splats. By introducing a self-supervised Variational AutoReconstructor, the model learns a rich 3D latent space directly from 2D images, enabling real-time, high-quality, and diverse 360° scene completions without requiring any 3D ground-truth data.

What Problem Does This Paper Address?

Reconstructing a full 3D scene from a single viewpoint is a notoriously difficult, ill-posed problem. The regions behind objects or outside the camera's field of view are completely hidden, leaving infinite possibilities for what the rest of the scene looks like.

Traditional "regression-based" methods (like Splatter Image) try to predict a single deterministic 3D output. Because they cannot choose between multiple valid possibilities for occluded areas, they end up predicting a blurry "average" of all potential shapes and textures. While generative models (like diffusion models) can capture these multiple distinct possibilities, existing 3D diffusion methods typically require expensive, hard-to-obtain 3D ground-truth data for training. The authors set out to solve this dilemma: how to generate sharp, diverse 3D scene completions from a single image while training purely on standard 2D image collections.

Methods

The authors break their pipeline down into two main stages:

1. Variational AutoReconstructor (VAR): To eliminate the need for 3D ground truth, the authors design VAR to learn a 3D latent space directly from 2D images. An encoder maps an input 2D image to a latent distribution. A reconstructor then decodes a sampled latent into a "Splatter Image" representation, which stores multiple layers of Gaussian splat parameters per pixel grid. This representation is then rendered into alternative target views using differentiable rendering and optimized via multi-view consistency losses (L2, SSIM, and LPIPS).
2. Latent Denoising Diffusion Model: Once the VAR defines a stable latent space, a denoising diffusion model is trained on these latents, conditioned on features from the input image. During inference, sampling different random noise vectors allows the model to generate multiple distinct, plausible 3D completions for the unobserved parts of the scene.

To preserve sharp details from the original view, the architecture incorporates Skip Connections from the encoder to the reconstructor. To prevent the model from "cheating"—meaning storing all the scene's information in the skip features rather than encoding the 3D structure in the latent space—they randomly zero out the skip features during training.

Expand

Expand

Datasets

The model was trained and evaluated on two prominent multi-view datasets:

• CO3D (Common Objects in 3D): The authors focus on the widely used Hydrants (723 scenes) and TeddyBears (1329 scenes) categories, which provide 360-degree real-world video captures of objects.
• RealEstate10K: A large-scale dataset consisting of video sequences of real-world indoor and outdoor residential environments.

Key Insights and Ideas

• Embracing Ambiguity with Generative 3D: Instead of forcing a network to guess a single "correct" hidden geometry (which causes blurriness), treating 3D reconstruction as a distribution allows the model to output sharp, distinct, and realistic alternative hypotheses.
• 3D Latents from 2D Supervision: Differentiable rendering acts as the mathematical bridge that allows a 3D latent space to be structured and regularized entirely by 2D multi-view consistency losses, bypassing the need for explicit 3D training data.
• The Skip Connection Trade-off: While skip connections are vital for preserving high-frequency textures from the input view, they can cause latent collapse if the network relies on them too heavily. Randomly masking them forces the latent space to capture the core 3D geometry.

Key Results

• State-of-the-Art Quality: On the CO3D dataset, the method outperforms existing real-time baselines in both structural fidelity (PSNR) and perceptual quality metrics (LPIPS, FID, KID).
• Blazing Fast Inference: Unlike NeRF-based diffusion models (such as DFM) which take around 10 minutes per scene due to computationally heavy per-frame denoising, this method generates the entire 3D Gaussian Splat scene in just 3 seconds.
• Multimodal Diversity: The model successfully generates diverse completions for occluded areas (e.g., adding different backside structures or textures to a hydrant) while keeping the visible parts strictly faithful to the input image.

Expand

Limitations and Future Work

• Frustum Constraints: Because the method builds on the Splatter Image framework, the underlying 3D representation is inherently tied to the camera frustum of a single reference view, making it difficult to scale to massive, unbounded 360° environments.
• Category Specificity: The model is currently trained on specific categories (e.g., hydrants or rooms). Scaling this up to a massive, open-world, multi-category foundation model remains an open challenge.
• Static Scenes Only: The framework assumes static environments. Extending this approach to 4D Gaussians to capture scene dynamics over time is a promising future direction noted by the authors.