TL;DR
MLLM-GS is a unified framework that leverages the rich visual priors of Multimodal Large Language Models (MLLMs) to perform high-quality novel view synthesis (NVS) in extreme environments like smoke, low light, and blur. By combining a lightweight correction network with a highly optimized 3D Gaussian Splatting engine, it achieves superior reconstruction quality while being significantly faster and more memory-efficient than existing state-of-the-art methods.
What Problem Does This Paper Address?
Reconstructing a 3D scene from a set of images—known as Novel View Synthesis (NVS)—is a cornerstone of computer vision. However, most existing methods assume the input images are clear and well-lit. In real-world "extreme environments," such as smoky rooms, low-light alleys, or scenes with heavy motion blur, these methods fall apart.
Current solutions usually rely on degradation-specific models (e.g., a specific "dehazing" model), which fail to generalize across different types of noise. While modern generative models can "hallucinate" clean versions of these images, their outputs often suffer from pixel misalignment and color shifts. When these slightly "off" images are used to build a 3D scene, the errors propagate, leading to structural distortions and visual artifacts. Furthermore, the high computational and memory costs of 3D reconstruction make it impractical for many real-world applications.
Methods
The authors propose MLLM-GS, a two-branch framework designed for both robustness and speed.
1. MLLM Prior-Guided View Enhancement
Instead of building a custom restorer for every type of degradation, the authors use a frozen Multimodal Large Language Model (MLLM)—specifically Nano Banana Pro—as a general-purpose visual prior. The MLLM is prompted to "restore a clear and sharp view" of the scene.
To fix the MLLM's tendency to introduce pixel shifts and color distortions, they introduce a lightweight correction network (only 4M parameters). This network operates in the HVI (Horizontal/Vertical-Intensity) color space, which decouples brightness from color, allowing for more effective correction. Crucially, this branch requires only a few (e.g., three) paired clean views to learn how to refine the MLLM's output into reliable "pseudo-GT" (Ground Truth) views for the rest of the scene.
2. Fast 3D Scene Reconstruction
Once the views are enhanced, the system uses 3D Gaussian Splatting (3DGS) to build the scene. To make this process "fast," the authors optimized three key areas:
- Rasterization: They replaced standard square bounding boxes with opacity-aware axis-aligned rectangular boxes, excluding regions that don't contribute to the image and saving massive amounts of computation.
- Memory Layout: They reordered Gaussian primitives using Morton encoding (a Z-order curve). This ensures that Gaussians that are physically near each other in 3D space are also near each other in the computer's memory, drastically improving cache efficiency.
- Parameter Updates: They fused multiple optimization steps (Adam updates) into a single CUDA kernel, reducing the overhead of launching multiple small tasks on the GPU.
Datasets
The framework was evaluated on the RealX3D benchmark, which contains real-world multi-view observations captured under challenging conditions:
- Smoke: Eight different scenes with varying haze density.
- Low-Light: Nine scenes with extreme underexposure and noise.
- Exposure & Blur: Additional tests on varying exposure, defocus blur, and motion blur.
Key Insights and Ideas
- MLLMs as Generalist Restorers: The most striking insight is that a general-purpose MLLM can outperform specialized restoration models when guided by a small amount of clean data. It acts as a "brain" that understands what a scene should look like, regardless of whether the interference is smoke or darkness.
- The HVI Advantage: By moving away from RGB space to HVI, the correction network can fix structural issues (luminance) without messing up the colors (chrominance), and vice versa.
- Efficiency through Locality: The use of Morton encoding highlights that 3D reconstruction speed isn't just about the number of points, but about how those points are organized in hardware memory.
Key Results
MLLM-GS sets new standards for both quality and efficiency in degraded environments:
- Quality: In smoke scenes, it achieved an average PSNR of 22.27 dB, a massive 4.69 dB improvement over the strongest restoration-based baseline.
- Speed: It completed 30,000 training iterations in roughly 10 minutes, offering a 9x speedup over some NeRF-based methods and a 1.46x speedup over standard 3DGS.
- Memory: It consumed only 1.98 GiB of VRAM for smoke scenes, compared to over 8 GiB for standard 3DGS, making it viable for consumer-grade hardware.
- Few-Shot Success: As shown in the ablation studies, the model reaches high performance (27 dB PSNR) with as few as three to five clean reference views.
Limitations and Future Work
While MLLM-GS is highly effective, the authors note that the quality of the final reconstruction is still tied to the quality of the underlying MLLM's generative prior. Different MLLMs yield different results, suggesting that as these models improve, MLLM-GS will naturally get better. The paper does not extensively discuss performance in highly dynamic scenes where objects are moving rapidly, which remains a common challenge for Gaussian Splatting techniques.
Summary updated