TL;DR
LiteReality is a novel pipeline that transforms noisy, incomplete indoor RGB-D scans into compact, interactive, and "graphics-ready" 3D virtual replicas. By combining training-free CAD model retrieval, LLM-guided contextual selection, and a lightweight material painting module, it produces simulation-ready environments that are fully compatible with standard graphics engines.
What Problem Does This Paper Address?
While modern 3D reconstruction techniques like Neural Radiance Fields (NeRFs) and 3D Gaussian Splatting achieve incredible visual photorealism, they essentially generate "3D photographs." Their outputs are static geometries or point clouds that lack semantic structure, object individuality, and physical interactivity—rendering them uneditable and incompatible with downstream applications like VR gaming, robotics simulation, or digital twins.
Existing CAD-retrieval methods try to solve this by replacing scanned objects with clean 3D models, but they often struggle with the clutter, severe occlusions, and poor lighting typical of real-world, "in-the-wild" scans. LiteReality bridges this gap by creating structured, object-centric, and physically interactive 3D scenes directly from raw, noisy scans.
Methods
LiteReality breaks the reconstruction process down into four main stages:
- Scene Perception and Parsing: Using off-the-shelf tools (such as Apple's RoomPlan), the pipeline extracts room layouts and 3D oriented bounding boxes. It organizes these into a structured scene graph and applies a Constraint-Based Collision Resolution algorithm to fix overlapping objects or floating items, ensuring physical plausibility.
- Object Reconstruction: Instead of training a complex network, the authors use a hierarchical, training-free retrieval pipeline. It filters candidates semantically, ranks them using DINOv2 visual features, performs pose-aware rendering comparisons, and uses a Multimodal Large Language Model (MLLM like GPT-4o) for final contextual selection based on style and proportion.
- Material Painting: To match the real-world appearance, this module maps cropped images to specific 3D model segments using SAM and Grounding DINO. It then performs a lightweight Albedo-Only Optimization in the CIE LAB color space. This shifts the global color distribution to match the photos while preserving high-frequency texture details, avoiding the massive computational cost of full differentiable PBR rendering.
- Procedural Reconstruction: Finally, the scene is assembled in Blender. The pipeline automatically constructs walls, doors, and windows, places the retrieved objects, and uses an MLLM to estimate object mass to enable realistic rigid-body physics simulations.
Datasets
- ScanNet: Used to evaluate object retrieval similarity on the Scan2CAD benchmark.
- LiteReality Database: A curated asset database containing 5,283 3D models sourced from 3D-FUTURE, AI2-THOR, and Sketchfab, complete with material-level segmentations and articulation data.
- New Benchmarks: The authors introduce two new benchmarks using real-world data captured via an iPhone 13 Pro Max:
- Object-Centric Material Recovery Benchmark: 111 objects across 5 indoor scenes to test material estimation under severe real-world occlusion and lighting challenges.
- Graphics-Ready Reconstruction Benchmark: Evaluates end-to-end full-scene reconstruction quality.
Key Insights and Ideas
- Graphics-Ready > Pure Photorealism: The core philosophy shifts focus from passive geometric reconstruction to creating functional, interactive assets (featuring articulated joints, correct physics properties, and PBR materials).
- MLLMs as Contextual Refiners: Purely visual embeddings (like DINOv2) can fail to distinguish stylistic nuances or proportions. Leveraging the reasoning capabilities of MLLMs for the final selection step drastically improves retrieval accuracy without requiring task-specific training.
- LAB Space Color Shifting: Instead of running expensive differentiable rendering loops to optimize complex SVBRDF parameters, simply shifting the mean albedo color in the CIE LAB space achieves visually accurate results at a fraction of the computational cost.
Key Results
- State-of-the-Art Retrieval: On the Scan2CAD benchmark, LiteReality achieved a two-way Chamfer Distance of 0.0986, outperforming existing methods like MSCD (0.1103) and Digital Cousin (0.1411).
- Superior Material Estimation: In the object-centric material benchmark, LiteReality outperformed baselines with a Root Mean Square Error (RMSE) of 0.2163 and a Structural Similarity Index (SSIM) of 0.4353.
- End-to-End Realism: For full-scene reconstruction, it significantly outperformed Phone2Proc and Digital Cousin across all perceptual metrics (RMSE 0.2664, SSIM 0.5818).
- Efficiency: Complete room-scale reconstruction takes between 20 to 60 minutes on a single NVIDIA RTX 3090 GPU, dominated mostly by the material painting stage.
Limitations and Future Work
- Upstream Dependency: The pipeline's quality is heavily bottlenecked by the initial layout and bounding box detection. Errors in the initial scan phase propagate downstream.
- Complex Collisions: The constraint-based collision solver can struggle in highly cluttered or densely packed environments, occasionally leading to object interpenetrations.
- Future Directions: The authors plan to extend the pipeline to detect smaller items, integrate learning-based light estimation for full SVBRDF recovery, scale up to multi-room/building-level environments, and optimize the system for real-time performance.
Summary updated