MuDG: Taming Multi-modal Diffusion with Gaussian Splatting for Urban Scene Reconstruction

Yingshuang Zou, Yikang Ding, Chuanrui Zhang, Jiazhe Guo, Bohan Li, Xiaoyang Lyu, Feiyang Tan, Xiaojuan Qi, Haoqian WangarXiv 2025 · DOI · OpenReview

TL;DR

This paper introduces MuDG, a novel framework that combines multi-modal diffusion models with Gaussian Splatting (GS) for robust 3D urban scene reconstruction and novel view synthesis (NVS). MuDG generates high-fidelity RGB, depth, and semantic outputs for unseen viewpoints without per-scene optimization, and these outputs are then used to significantly improve the training of GS models for better performance under extreme camera movements.

What Problem Does This Paper Address?

Reconstructing and rendering dynamic urban scenes for applications like autonomous driving is challenging. Existing methods fall into two camps: reconstruction-based approaches (like NeRF and 3DGS) struggle with significant viewpoint deviations from their training data, leading to artifacts. Generative methods, while capable of producing novel views, often lack temporal coherence and precise control over the scene. This gap means current methods are insufficient for tasks requiring flexible viewpoint control or reliable synthetic data generation for corner-case scenarios in autonomous driving.

Methods

MuDG tackles this by integrating two key components:

  1. Multi-modal Diffusion Model (MDM): This is the core generative part. It takes sparse RGB and depth information derived from aggregated LiDAR point clouds as conditional inputs. The model is trained to synthesize dense RGB, depth, and semantic maps for novel viewpoints. A key aspect is its "feed-forward" nature, meaning it doesn't require computationally intensive per-scene optimization for new scenarios. It leverages a pre-trained VAE for encoding/decoding multi-modal data into a latent space and a diffusion model for generating the outputs.
  2. Gaussian Splatting (GS) Module: The outputs from the MDM (RGB, depth, and semantic maps) are used as supervisory signals to train a 3D Gaussian Splatting representation. This is crucial because the MDM's generated data provides comprehensive supervision, enabling the GS model to achieve robust rendering quality even when facing extreme viewpoint changes, thereby overcoming a major limitation of standard GS.

The framework aggregates LiDAR point clouds, reprojects them into sparse color and depth maps, and uses these, along with a reference image, as conditional inputs for the MDM. For GS training, it leverages virtual viewpoints generated by the MDM to improve geometric consistency and temporal stability.

Datasets

The primary dataset used for training and evaluation is the Open Waymo Dataset (WOD). For semantic map ground truth, the paper mentions using the [31] dataset. For depth, they aggregate LiDAR frames and project them to create sparse depth maps.

Key Insights and Ideas

  • Synergy of Diffusion and GS: The paper demonstrates that combining the generative power of diffusion models with the explicit scene representation of Gaussian Splatting is highly effective for dynamic urban scenes. Diffusion models provide flexibility and controllability, while GS offers high-fidelity rendering.
  • Multi-modal Supervision is Key: Using synthesized RGB, depth, and semantic information from the diffusion model to supervise GS training significantly boosts performance, especially in challenging dynamic urban environments. This multi-modal supervision bridges the gap between synthetic and real data.
  • Feed-Forward NVS for Control: The MDM's ability to generate novel views in a feed-forward manner, without per-scene optimization, is a significant step towards practical applications requiring real-time control and editing capabilities.
  • Robustness to Viewpoint Changes: The proposed approach shows remarkable stability and quality preservation even under large lateral shifts (e.g., 4 meters), outperforming existing methods that degrade significantly in such scenarios.

Key Results

  • State-of-the-Art Performance: MuDG outperforms existing methods on the Open Waymo Dataset in both reconstruction and synthesis quality, as measured by metrics like FID and FVD.
  • Quantitative Improvements: The paper reports significant improvements in metrics like FID and FVD compared to baseline methods, particularly for out-of-trajectory viewpoints. For instance, incorporating virtual view color information reduced FID by 18.7% and improved FVD by 23.3%.
  • Qualitative Superiority: Visualizations show MuDG producing photorealistic and consistent multi-modal outputs (RGB, depth, semantic) for novel views, even with significant perspective shifts. The editing capabilities (background changes, object removal) are also demonstrated to maintain consistency.
  • Enhanced GS: The integration with GS leads to better geometric quality and structural preservation under extreme viewpoint offsets.

Limitations and Future Work

  • Computational Cost: The authors acknowledge that both training and inference for diffusion models are computationally intensive, leading to substantial time costs. Accelerating the model is identified as a key area for future improvement.
  • Pseudo Ground Truth Dependency: The training process relies on pre-processed pseudo ground truth data (depth and semantic maps), which adds to the overall time cost. Optimizing the generation of these pseudo GTs is suggested as a future focus.

Summary updated

Abstract

Recent breakthroughs in radiance fields have significantly advanced 3D scene reconstruction and novel view synthesis (NVS) in autonomous driving. Nevertheless, critical limitations persist: reconstruction-based methods exhibit substantial performance deterioration under significant viewpoint deviations from training trajectories, while generation-based techniques struggle with temporal coherence and precise scene controllability. To overcome these challenges, we present MuDG, an innovative framework that integrates Multi-modal Diffusion model with Gaussian Splatting (GS) for Urban Scene Reconstruction. MuDG leverages aggregated LiDAR point clouds with RGB and geometric priors to condition a multi-modal video diffusion model, synthesizing photorealistic RGB, depth, and semantic outputs for novel viewpoints. This synthesis pipeline enables feed-forward NVS without computationally intensive per-scene optimization, providing comprehensive supervision signals to refine 3DGS representations for rendering robustness enhancement under extreme viewpoint changes. Experiments on the Open Waymo Dataset demonstrate that MuDG outperforms existing methods in both reconstruction and synthesis quality.
BibTeX
@misc{zou2025mudg,
  author = {Yingshuang Zou and Yikang Ding and Chuanrui Zhang and Jiazhe Guo and Bohan Li and Xiaoyang Lyu and Feiyang Tan and Xiaojuan Qi and Haoqian Wang},
  title = {{MuDG: Taming Multi-modal Diffusion with Gaussian Splatting for Urban Scene Reconstruction}},
  year = {2025},
  eprint = {2503.10604},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.10604},
}
Figure