LiteReality: Graphics-Ready 3D Scene Reconstruction from RGB-D Scans

Zhening Huang, Xiaoyang Wu, Fangcheng Zhong, Hengshuang Zhao, Matthias Nießner, Joan Lasenby

TL;DR

LiteReality is a novel pipeline that transforms noisy, incomplete indoor RGB-D scans into compact, interactive, and "graphics-ready" 3D virtual replicas. By combining training-free CAD model retrieval, LLM-guided contextual selection, and a lightweight material painting module, it produces simulation-ready environments that are fully compatible with standard graphics engines.

What Problem Does This Paper Address?

While modern 3D reconstruction techniques like Neural Radiance Fields (NeRFs) and 3D Gaussian Splatting achieve incredible visual photorealism, they essentially generate "3D photographs." Their outputs are static geometries or point clouds that lack semantic structure, object individuality, and physical interactivity—rendering them uneditable and incompatible with downstream applications like VR gaming, robotics simulation, or digital twins.

Existing CAD-retrieval methods try to solve this by replacing scanned objects with clean 3D models, but they often struggle with the clutter, severe occlusions, and poor lighting typical of real-world, "in-the-wild" scans. LiteReality bridges this gap by creating structured, object-centric, and physically interactive 3D scenes directly from raw, noisy scans.

Methods

LiteReality breaks the reconstruction process down into four main stages:

  1. Scene Perception and Parsing: Using off-the-shelf tools (such as Apple's RoomPlan), the pipeline extracts room layouts and 3D oriented bounding boxes. It organizes these into a structured scene graph and applies a Constraint-Based Collision Resolution algorithm to fix overlapping objects or floating items, ensuring physical plausibility.
  2. Object Reconstruction: Instead of training a complex network, the authors use a hierarchical, training-free retrieval pipeline. It filters candidates semantically, ranks them using DINOv2 visual features, performs pose-aware rendering comparisons, and uses a Multimodal Large Language Model (MLLM like GPT-4o) for final contextual selection based on style and proportion.
  3. Material Painting: To match the real-world appearance, this module maps cropped images to specific 3D model segments using SAM and Grounding DINO. It then performs a lightweight Albedo-Only Optimization in the CIE LAB color space. This shifts the global color distribution to match the photos while preserving high-frequency texture details, avoiding the massive computational cost of full differentiable PBR rendering.
  4. Procedural Reconstruction: Finally, the scene is assembled in Blender. The pipeline automatically constructs walls, doors, and windows, places the retrieved objects, and uses an MLLM to estimate object mass to enable realistic rigid-body physics simulations.
Figure 2: Pipeline of LiteReality. With input RGB-D scans, the process begins with scene perception and parsing, where room layouts and 3D object bounding box are detected and parsed into a structured and physically plausible arrangement with the help of a scene graph. In the object reconstruction stage, identical objects are first grouped into clusters so that 3D model retrieval from the LiteReality database is performed once per cluster if possible. The material painting stage retrieves and optimizes PBR materials by referencing observed images. Finally, procedural reconstruction assembles all the information into a graphics-ready environment that features realism and interactivity.

Datasets

  • ScanNet: Used to evaluate object retrieval similarity on the Scan2CAD benchmark.
  • LiteReality Database: A curated asset database containing 5,283 3D models sourced from 3D-FUTURE, AI2-THOR, and Sketchfab, complete with material-level segmentations and articulation data.
  • New Benchmarks: The authors introduce two new benchmarks using real-world data captured via an iPhone 13 Pro Max:
  • Object-Centric Material Recovery Benchmark: 111 objects across 5 indoor scenes to test material estimation under severe real-world occlusion and lighting challenges.
  • Graphics-Ready Reconstruction Benchmark: Evaluates end-to-end full-scene reconstruction quality.

Key Insights and Ideas

  • Graphics-Ready > Pure Photorealism: The core philosophy shifts focus from passive geometric reconstruction to creating functional, interactive assets (featuring articulated joints, correct physics properties, and PBR materials).
  • MLLMs as Contextual Refiners: Purely visual embeddings (like DINOv2) can fail to distinguish stylistic nuances or proportions. Leveraging the reasoning capabilities of MLLMs for the final selection step drastically improves retrieval accuracy without requiring task-specific training.
  • LAB Space Color Shifting: Instead of running expensive differentiable rendering loops to optimize complex SVBRDF parameters, simply shifting the mean albedo color in the CIE LAB space achieves visually accurate results at a fraction of the computational cost.

Key Results

  • State-of-the-Art Retrieval: On the Scan2CAD benchmark, LiteReality achieved a two-way Chamfer Distance of 0.0986, outperforming existing methods like MSCD (0.1103) and Digital Cousin (0.1411).
  • Superior Material Estimation: In the object-centric material benchmark, LiteReality outperformed baselines with a Root Mean Square Error (RMSE) of 0.2163 and a Structural Similarity Index (SSIM) of 0.4353.
  • End-to-End Realism: For full-scene reconstruction, it significantly outperformed Phone2Proc and Digital Cousin across all perceptual metrics (RMSE 0.2664, SSIM 0.5818).
  • Efficiency: Complete room-scale reconstruction takes between 20 to 60 minutes on a single NVIDIA RTX 3090 GPU, dominated mostly by the material painting stage.
Figure 1: From RGB-D Scan to Graphics-Ready 3D Scene. LiteReality reconstructs compact, realistic 3D environments from real-world RGB-D scans, featuring near-photorealistic appearance, articulated geometry, and physically based rendering (PBR) materials—providing assets that can be easily integrated into simulation or rendering pipeline.

Limitations and Future Work

  • Upstream Dependency: The pipeline's quality is heavily bottlenecked by the initial layout and bounding box detection. Errors in the initial scan phase propagate downstream.
  • Complex Collisions: The constraint-based collision solver can struggle in highly cluttered or densely packed environments, occasionally leading to object interpenetrations.
  • Future Directions: The authors plan to extend the pipeline to detect smaller items, integrate learning-based light estimation for full SVBRDF recovery, scale up to multi-room/building-level environments, and optimize the system for real-time performance.

Summary updated

Abstract

We propose LiteReality, a novel pipeline that converts RGB-D scans of indoor environments into compact, realistic, and interactive 3D virtual replicas. LiteReality not only reconstructs scenes that visually resemble reality but also supports key features essential for graphics pipelines -- such as object individuality, articulation, high-quality physically based rendering materials, and physically based interaction. At its core, LiteReality first performs scene understanding and parses the results into a coherent 3D layout and objects with the help of a structured scene graph. It then reconstructs the scene by retrieving the most visually similar 3D artist-crafted models from a curated asset database. Next, the Material Painting module enhances realism by recovering high-quality, spatially varying materials. Finally, the reconstructed scene is integrated into a simulation engine with basic physical properties to enable interactive behavior. The resulting scenes are compact, editable, and fully compatible with standard graphics pipelines, making them suitable for applications in AR/VR, gaming, robotics, and digital twins. In addition, LiteReality introduces a training-free object retrieval module that achieves state-of-the-art similarity performance on the Scan2CAD benchmark, along with a robust material painting module capable of transferring appearances from images of any style to 3D assets -- even under severe misalignment, occlusion, and poor lighting. We demonstrate the effectiveness of LiteReality on both real-life scans and public datasets. Project page: https://litereality.github.io; Video: https://www.youtube.com/watch?v=ecK9m3LXg2c
BibTeX
@inproceedings{huang2025litereality,
  author = {Zhening Huang and Xiaoyang Wu and Fangcheng Zhong and Hengshuang Zhao and Matthias Nießner and Joan Lasenby},
  title = {{LiteReality: Graphics-Ready 3D Scene Reconstruction from RGB-D Scans}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=4JLZsmWBJf},
}
Figure