3D Scene Rendering with Multimodal Gaussian Splatting

Chi-Shiang Gau, Konstantinos D. Polyzos, Athanasios Bacharis, Saketh Madhuvarasu, Tara JavidiarXiv 2026 · arXiv

TL;DR

This paper introduces a multimodal framework that uses sparse radio-frequency (RF) signals, like automotive radar, to jumpstart 3D Gaussian Splatting (GS). By employing a novel "localized Gaussian Process" to reconstruct dense depth maps from sparse radar data, the authors achieve significantly faster initialization and higher rendering quality than traditional vision-only methods, especially in scenarios with limited camera views.

What Problem Does This Paper Address?

3D Gaussian Splatting has revolutionized real-time scene rendering, but it has a "cold start" problem. To work well, GS needs a high-quality initial 3D point cloud to place its Gaussian primitives. Traditionally, this is done using Structure-from-Motion (SfM) tools like COLMAP, which require many overlapping images and significant processing time.

This vision-only approach hits a wall in two major ways:

  1. Environmental Sensitivity: Cameras struggle in adverse weather (fog, rain), low light, or scenes with heavy occlusions.
  2. Efficiency: SfM is computationally expensive and slow, making it poorly suited for real-time robotics or autonomous driving applications where a scene needs to be understood quickly from just a few snapshots.

The authors propose using radar as a complementary modality because RF signals are virtually immune to lighting and weather conditions.

Methods

The core of the paper is a multimodal pipeline that replaces traditional vision-based initialization with an RF-driven one.

  • Localized Gaussian Processes (GPs): Standard GPs are great for predicting values (like depth) at unobserved locations, but they suffer from $O(T^3)$ computational complexity, making them slow for large datasets. The authors introduce a localization scheme that partitions the scene into non-overlapping regions, each handled by its own local GP. This not only slashes computation time but also prevents distant, irrelevant data points from "blurring" local depth predictions.
  • RF-to-Point Cloud: The system takes sparse depth measurements from a single radar transmission (see Fig 1) and uses the localized GP to predict a dense depth map.
  • Uncertainty Filtering: Because GPs provide variance (uncertainty) estimates, the framework can intelligently discard low-confidence depth predictions, ensuring the resulting 3D point cloud is reliable.
  • GS Optimization: Once initialized with this RF-informed point cloud, the Gaussian parameters (color, opacity, etc.) are optimized using standard GS techniques against the available training images.
Fig. 1: Initial depth map from a single radar transmission consisting of sparse depth measurements/observations.

Datasets

The authors evaluated their approach using the View-of-Delft (VoD) dataset. This is a real-world urban driving dataset that includes synchronized RGB camera images and 3+1D radar scans. For their experiments, they used a challenging "sparse-view" setup: only 12 images for training and 23 for testing.

Key Insights and Ideas

  • Radar as a Structural "Cheat Code": While radar data is sparse and noisy, it provides a robust geometric skeleton that vision-only methods struggle to find when images are few or poor quality.
  • The Power of Localization: Moving from a global GP to localized GPs isn't just a speed hack; it actually improves accuracy. By focusing on spatially proximate observations, the model captures local structural nuances better than a global model that tries to fit the entire scene at once.
  • Computational Efficiency for Robotics: The transition from minutes (COLMAP) to seconds (RF-GP) for initialization is a game-changer for autonomous agents that need to render and understand new environments on the fly.

Key Results

The multimodal approach showed clear advantages in both speed and quality:

  • Initialization Speed: Generating the initial point cloud took approximately 0.81 seconds using the localized GP, compared to 4.43 minutes for the vision-only COLMAP baseline.
  • Depth Accuracy: The localized GP achieved a Mean Absolute Error (MAE) of 10.57m, significantly better than the 13.07m MAE of a conventional global GP.
  • Rendering Quality: In sparse-view tests, the multimodal GS significantly outperformed vision-only GS across all standard metrics:
  • PSNR: 15.032 (Ours) vs. 13.339 (Vision-only)
    • SSIM: 0.4628 (Ours) vs. 0.4161 (Vision-only)
    • LPIPS: 0.4727 (Ours) vs. 0.5114 (Vision-only)

Qualitatively, the rendered images from the multimodal approach show much sharper structures and fewer artifacts compared to the blurry results of the unimodal baseline (see Fig 5).

Fig 5

Limitations and Future Work

While promising, the authors note a few areas for growth:

  • Single Transmission: The current tests use only a single radar transmission. Integrating multiple transmissions over time (temporal fusion) could further improve the initial point cloud density.
  • Sensor Fusion Expansion: The framework is designed to be modular. The authors suggest that incorporating LiDAR—when available—alongside radar and vision could create an even more robust "tri-modal" system.
  • Complex Occlusions: While radar handles partial occlusions well, extremely complex urban geometries still pose challenges for sparse depth reconstruction.

Summary updated

Abstract

3D scene reconstruction and rendering are core tasks in computer vision, with applications spanning industrial monitoring, robotics, and autonomous driving. Recent advances in 3D Gaussian Splatting (GS) and its variants have achieved impressive rendering fidelity while maintaining high computational and memory efficiency. However, conventional vision-based GS pipelines typically rely on a sufficient number of camera views to initialize the Gaussian primitives and train their parameters, typically incurring additional processing cost during initialization while falling short in conditions where visual cues are unreliable, such as adverse weather, low illumination, or partial occlusions. To cope with these challenges, and motivated by the robustness of radio-frequency (RF) signals to weather, lighting, and occlusions, we introduce a multimodal framework that integrates RF sensing, such as automotive radar, with GS-based rendering as a more efficient and robust alternative to vision-only GS rendering. The proposed approach enables efficient depth prediction from only sparse RF-based depth measurements, yielding a high-quality 3D point cloud for initializing Gaussian functions across diverse GS architectures. Numerical tests demonstrate the merits of judiciously incorporating RF sensing into GS pipelines, achieving high-fidelity 3D scene rendering driven by RF-informed structural accuracy.
BibTeX
@misc{gau2026d,
  author = {Chi-Shiang Gau and Konstantinos D. Polyzos and Athanasios Bacharis and Saketh Madhuvarasu and Tara Javidi},
  title = {{3D Scene Rendering with Multimodal Gaussian Splatting}},
  year = {2026},
  eprint = {2602.17124},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.17124},
}
Figure