GaussFusion: Improving 3D Reconstruction in the Wild with A Geometry-Informed Video Generator

Liyuan Zhu, Manjunath Narayana, Michal Stary, Will Hutchcroft, Gordon Wetzstein, Iro Armeni

TL;DR

GaussFusion introduces a geometry-informed video-to-video generative framework designed to remove persistent artifacts—like floaters, blur, and flickering—from 3D Gaussian Splatting (3DGS) reconstructions. By conditioning a video diffusion model on a multi-modal "GP-Buffer" (containing depth, normals, and uncertainty), it generalizes across both optimization-based and feed-forward 3DGS pipelines. A distilled variant achieves real-time performance at 15+ FPS while delivering state-of-the-art rendering quality.

What Problem Does This Paper Address?

3D Gaussian Splatting (3DGS) has revolutionized photorealistic 3D reconstruction and novel-view synthesis due to its high rendering speed and explicit scene representation. However, when dealing with sparse views, incomplete coverage, or camera pose errors, 3DGS models frequently suffer from severe visual artifacts. These include "floaters" (detached, blurry clusters of Gaussians), texture distortions, and temporal flickering.

While recent approaches try to fix these issues using generative image or video diffusion models as a post-hoc cleanup step, they typically rely solely on RGB or basic RGB-D inputs. This limited conditioning creates ambiguity; an RGB-only model cannot easily distinguish between a true geometry error (like a floater) and a complex texture. Furthermore, existing enhancement methods are usually overfitted to a single reconstruction paradigm—either optimization-based or feed-forward—failing to generalize across different 3DGS backbones.

Methods

GaussFusion solves these limitations by introducing a geometry-aware video-to-video generation pipeline.

GaussFusion Overview

The core architecture consists of three main components:

  1. Gaussian Primitives Buffer (GP-Buffer): Instead of rendering just RGB images, GaussFusion extracts a pixel-aligned, multi-modal video buffer from the underlying 3DGS representation. The GP-Buffer encodes five distinct modalities: Color ($C$), Opacity ($A$), Expected Depth ($D$), Screen-space Surface Normals ($N$), and Geometric Uncertainty ($U$, derived from the inverse projected 2D covariance of the splats).
  2. Geometry-Informed Video Generator: Built on top of the Wan-2.1 Diffusion Transformer (DiT) backbone, the model processes noised latents alongside the encoded GP-Buffer features. A novel Geometry Adapter (GA) module injects these multi-modal geometric tokens into the DiT blocks via self-attention and cross-attention mechanisms, ensuring tight spatial alignment between the geometry and the generated video frames.
  3. Two-Stage Finetuning Strategy: To make the model practical for interactive applications, the authors apply Distribution Matching Distillation (DMD) to compress the typical 30–50 denoising steps of a video diffusion model down to an efficient 4-step generation process.
GaussFusion Video Generator Architecture

Datasets

The model was trained and evaluated using two prominent novel-view synthesis benchmarks:

  • DL3DV: A large-scale scene dataset for deep learning-based 3D vision.
  • RE10K (RealEstate10K): A dataset consisting of camera trajectories from real estate videos.

To train a robust refiner, the authors built an Artifact Simulation Pipeline that generated over 75,000 paired video samples (clean ground-truth vs. corrupted renderings) from these datasets. The corruptions realistically mimic real-world defects by downsampling training views to 5%, varying initialization strategies (SfM vs. random point clouds), and utilizing flawed feed-forward model predictions.

Key Insights and Ideas

  • Geometry Disambiguates Artifacts: Relying on RGB alone makes it difficult to clear out large floaters or fill missing regions. Adding explicit geometric cues like surface normals and inverse covariance allows the network to "see through" artifacts and accurately repaint missing or distorted structures.
  • Cross-Paradigm Artifact Differences: Optimization-based 3DGS tends to fail with high-frequency noise ("needles" and floaters due to overfitting), whereas feed-forward models suffer from low-frequency blur and global warping due to regression uncertainty. Training on a mixed dataset containing both degradation types is essential for creating a universally applicable refiner.
  • 3DGS as an Explicit Memory: Pure video generation models often suffer from temporal drift because they lack a 3D anchor. GaussFusion uses the 3DGS model as an explicit 3D memory, enforcing mathematically correct projections via rasterization and preventing geometric drift.

Key Results

GaussFusion achieves state-of-the-art performance across both optimization-based and feed-forward 3DGS methods.

Quantitative Performance

On the RE10K dataset, GaussFusion significantly boosted the rendering quality of initial reconstructions:

  • For Splatfacto (optimization-based), it raised the PSNR from 19.23 to 28.65 and slashed the FID score from 12.62 to 6.67.
  • For DepthSplat (feed-forward), it improved the PSNR from 21.77 to 22.80.

Crucially, the distilled Few-step variant maintained high fidelity (28.25 PSNR on RE10K) while running at 15.11 FPS, making real-time, on-the-fly refinement possible.

Qualitative Improvements

Ablation studies highlighted the power of the GP-Buffer over RGB-only alternatives. As shown below, incorporating the full suite of geometric attributes helps the model eliminate floaters and resolve fine details that RGB-only models smear out.

Qualitative Ablation on GP-Buffer

Limitations and Future Work

Despite its strong performance, GaussFusion has a few notable limitations:

  • Rapid Motion & Motion Blur: The underlying VAE struggles to encode clean latent representations when videos contain severe motion blur or extremely rapid camera movements, leading to degraded outputs.
  • Long-Term Coherence: The model operates on a sliding context window of 81 frames. While it maintains excellent short-term consistency, it does not fully guarantee long-term coherence across very long video sequences.

Future Directions: The authors suggest exploring advanced distillation techniques like self-forcing and incorporating causal modeling with a rolling KV cache to better support streaming. Additionally, they aim to develop mechanisms that allow the video generator to directly update the 3D Gaussian parameters online, removing the need for post-hoc optimization entirely.

Summary updated

Abstract

We present GaussFusion, a novel approach for improving 3D Gaussian splatting (3DGS) reconstructions in the wild through geometry-informed video generation. GaussFusion mitigates common 3DGS artifacts, including floaters, flickering, and blur caused by camera pose errors, incomplete coverage, and noisy geometry initialization. Unlike prior RGB-based approaches limited to a single reconstruction pipeline, our method introduces a geometry-informed video-to-video generator that refines 3DGS renderings across both optimization-based and feed-forward methods. Given an existing reconstruction, we render a Gaussian primitive video buffer encoding depth, normals, opacity, and covariance, which the generator refines to produce temporally coherent, artifact-free frames. We further introduce an artifact synthesis pipeline that simulates diverse degradation patterns, ensuring robustness and generalization. GaussFusion achieves state-of-the-art performance on novel-view synthesis benchmarks, and an efficient variant runs in real time at 15 FPS while maintaining similar performance, enabling interactive 3D applications.
BibTeX
@inproceedings{zhu2026gaussfusion,
  author = {Liyuan Zhu and Manjunath Narayana and Michal Stary and Will Hutchcroft and Gordon Wetzstein and Iro Armeni},
  title = {{GaussFusion: Improving 3D Reconstruction in the Wild with A Geometry-Informed Video Generator}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2026},
  url = {https://openreview.net/forum?id=LOePXGz9Z2},
}
Figure