MLLM-GS: Multimodal Large Language Model Prior-Guided Fast Novel View Synthesis in Extreme Environments

Zhiliang Wu, Xinye Zheng, Kun Li, Fei Wang, Hao Guo, Yanyan Wei, Guanghui Yue, Beier Zhu, Wei Zhou, Xun YangarXiv 2026 · arXiv

TL;DR

MLLM-GS is a unified framework that leverages the rich visual priors of Multimodal Large Language Models (MLLMs) to perform high-quality novel view synthesis (NVS) in extreme environments like smoke, low light, and blur. By combining a lightweight correction network with a highly optimized 3D Gaussian Splatting engine, it achieves superior reconstruction quality while being significantly faster and more memory-efficient than existing state-of-the-art methods.

What Problem Does This Paper Address?

Reconstructing a 3D scene from a set of images—known as Novel View Synthesis (NVS)—is a cornerstone of computer vision. However, most existing methods assume the input images are clear and well-lit. In real-world "extreme environments," such as smoky rooms, low-light alleys, or scenes with heavy motion blur, these methods fall apart.

Current solutions usually rely on degradation-specific models (e.g., a specific "dehazing" model), which fail to generalize across different types of noise. While modern generative models can "hallucinate" clean versions of these images, their outputs often suffer from pixel misalignment and color shifts. When these slightly "off" images are used to build a 3D scene, the errors propagate, leading to structural distortions and visual artifacts. Furthermore, the high computational and memory costs of 3D reconstruction make it impractical for many real-world applications.

Methods

The authors propose MLLM-GS, a two-branch framework designed for both robustness and speed.

1. MLLM Prior-Guided View Enhancement

Instead of building a custom restorer for every type of degradation, the authors use a frozen Multimodal Large Language Model (MLLM)—specifically Nano Banana Pro—as a general-purpose visual prior. The MLLM is prompted to "restore a clear and sharp view" of the scene.

To fix the MLLM's tendency to introduce pixel shifts and color distortions, they introduce a lightweight correction network (only 4M parameters). This network operates in the HVI (Horizontal/Vertical-Intensity) color space, which decouples brightness from color, allowing for more effective correction. Crucially, this branch requires only a few (e.g., three) paired clean views to learn how to refine the MLLM's output into reliable "pseudo-GT" (Ground Truth) views for the rest of the scene.

2. Fast 3D Scene Reconstruction

Once the views are enhanced, the system uses 3D Gaussian Splatting (3DGS) to build the scene. To make this process "fast," the authors optimized three key areas:

  • Rasterization: They replaced standard square bounding boxes with opacity-aware axis-aligned rectangular boxes, excluding regions that don't contribute to the image and saving massive amounts of computation.
  • Memory Layout: They reordered Gaussian primitives using Morton encoding (a Z-order curve). This ensures that Gaussians that are physically near each other in 3D space are also near each other in the computer's memory, drastically improving cache efficiency.
  • Parameter Updates: They fused multiple optimization steps (Adam updates) into a single CUDA kernel, reducing the overhead of launching multiple small tasks on the GPU.
Fig. 2: Overall pipeline of the proposed MLLM-GS framework. It consists of two branches: MLLM prior-guided view enhancement and fast 3D scene reconstruction. The former combines a frozen MLLM with a lightweight correction network trained using a few paired clean views to generate reliable pseudo-GT views for subsequent 3D reconstruction. The latter accelerates 3DGS optimization through efficient rasterization, memory layout, and parameter updates, enabling fast and high-quality NVS.

Datasets

The framework was evaluated on the RealX3D benchmark, which contains real-world multi-view observations captured under challenging conditions:

  • Smoke: Eight different scenes with varying haze density.
  • Low-Light: Nine scenes with extreme underexposure and noise.
  • Exposure & Blur: Additional tests on varying exposure, defocus blur, and motion blur.

Key Insights and Ideas

  • MLLMs as Generalist Restorers: The most striking insight is that a general-purpose MLLM can outperform specialized restoration models when guided by a small amount of clean data. It acts as a "brain" that understands what a scene should look like, regardless of whether the interference is smoke or darkness.
  • The HVI Advantage: By moving away from RGB space to HVI, the correction network can fix structural issues (luminance) without messing up the colors (chrominance), and vice versa.
  • Efficiency through Locality: The use of Morton encoding highlights that 3D reconstruction speed isn't just about the number of points, but about how those points are organized in hardware memory.

Key Results

MLLM-GS sets new standards for both quality and efficiency in degraded environments:

  • Quality: In smoke scenes, it achieved an average PSNR of 22.27 dB, a massive 4.69 dB improvement over the strongest restoration-based baseline.
  • Speed: It completed 30,000 training iterations in roughly 10 minutes, offering a 9x speedup over some NeRF-based methods and a 1.46x speedup over standard 3DGS.
  • Memory: It consumed only 1.98 GiB of VRAM for smoke scenes, compared to over 8 GiB for standard 3DGS, making it viable for consumer-grade hardware.
  • Few-Shot Success: As shown in the ablation studies, the model reaches high performance (27 dB PSNR) with as few as three to five clean reference views.
Comparison of efficiency (PSNR, Time, VRAM) against other methods.

Limitations and Future Work

While MLLM-GS is highly effective, the authors note that the quality of the final reconstruction is still tied to the quality of the underlying MLLM's generative prior. Different MLLMs yield different results, suggesting that as these models improve, MLLM-GS will naturally get better. The paper does not extensively discuss performance in highly dynamic scenes where objects are moving rapidly, which remains a common challenge for Gaussian Splatting techniques.

Summary updated

Abstract

Novel view synthesis (NVS) in degraded scenes aims to reconstruct 3D scenes from multi-view observations affected by various degradations, and to generate clear and photorealistic images at unseen viewpoints. Despite recent advances, existing methods often rely on degradation-specific models, limiting their generalization across diverse extreme environments. Generative models provide rich visual priors, but their outputs often suffer from pixel misalignment and color deviations, which can propagate into the reconstructed 3D scene and compromise reconstruction quality. Furthermore, the high computational and memory costs of 3D reconstruction hinder the practical application of NVS in extreme environments. To this end, we propose MLLM-GS, a multimodal large language model (MLLM) prior-guided framework for fast NVS in extreme environments. MLLM-GS consists of two branches: MLLM prior-guided view enhancement and fast 3D scene reconstruction. The former leverages the rich visual priors of a frozen MLLM to enhance views under different severe degradations and introduces a lightweight correction network to reduce pixel shifts and color distortions in MLLM outputs. With supervision from only a few paired clean views, this branch produces reliable enhanced views for all degraded observations, providing pseudo-GT views for subsequent 3D reconstruction. The latter improves 3DGS efficiency by optimizing rasterization, Gaussian memory layout, and parameter updates, reducing redundant computation, memory-access overhead, and optimization costs. Extensive experiments on multiple degraded scenes demonstrate that MLLM-GS effectively handles various severe degradations and achieves high-quality NVS with significantly lower reconstruction time and memory consumption. Project Page: https://wzlwlm13.github.io/MLLM-GS
BibTeX
@misc{wu2026mllm,
  author = {Zhiliang Wu and Xinye Zheng and Kun Li and Fei Wang and Hao Guo and Yanyan Wei and Guanghui Yue and Beier Zhu and Wei Zhou and Xun Yang},
  title = {{MLLM-GS: Multimodal Large Language Model Prior-Guided Fast Novel View Synthesis in Extreme Environments}},
  year = {2026},
  eprint = {2610.39936},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.39936},
}
Figure