TL;DR
The authors introduce Lumera, a novel pipeline and benchmark designed to reconstruct fully editable 3D scenes—complete with discrete object meshes and parametric light sources—from a single image directly into standard game engines like Unreal Engine 5 and Blender. By framing 3D reconstruction as structured entity parsing rather than continuous radiance fields, Lumera bridges the gap between AI generation and practical digital content creation.
What Problem Does This Paper Address?
While modern 3D generative AI (like NeRFs or 3D Gaussian Splatting) can turn a single image into a stunning 3D view, the resulting outputs are often "baked." The geometry is a fused, uneditable blob, and the lighting is permanently cooked into the textures.
For game developers, animators, and VFX artists, these outputs are practically useless. Production pipelines require editable assets: individual object instances that can be moved or swapped, and actual, countable light fixtures with adjustable intensities, colors, and positions. Lumera addresses this gap by treating single-image reconstruction as an engine-native structured parsing problem, outputting discrete, inspectable entities instead of continuous volumes.
Methods
The Lumera pipeline breaks down the complex task of scene reconstruction into a multi-stage, entity-centric framework:
- Geometry Front-End: The pipeline uses Depth Anything 3 to generate a camera-aligned, colored point cloud from a single input image.
- Decoupled VLM Parsing: Instead of predicting everything at once, Lumera splits the workload into two independent Vision-Language Models (VLMs) based on the SpatialLM architecture:
- Lumera-Box: Parses the point cloud to emit oriented 3D bounding boxes and semantic labels.
- Lumera-Light: Emits parametric light tuples containing 3D coordinates, RGB color values, and intensity parameters $(x, y, z, r, g, b, I)$.
- Mesh & Environment Assembly: Bounding boxes guide instance segmentation masks (via SAM) to extract object crops, which are reconstructed into textured meshes using SAM3D. Global illumination is captured by estimating an HDR environment probe via IntrinsicHDR.
- Bounded Agentic Refinement: To fix residual errors in alignment or lighting, the scene is passed to a GPT-4-driven Generator/Verifier loop. Crucially, this loop is stage-aware: it refines geometry (yaw and scale) and lighting in entirely separate passes, utilizing strict whitelists and automated rollbacks to prevent the model from hallucinating or breaking the scene structure.
Datasets
To train and evaluate this light-aware pipeline, the authors introduced Lumera-2K, a massive game-engine dataset derived from 2,513 Unreal Engine 5 projects. It includes:
- 3.73M components and 63M object instances.
- 95.1K diverse camera views generated via an automated camera-planning pipeline.
- 102.6K engine-native parametric lights (Point, Spot, Rect, and Directional lights).
Unlike standard 3D datasets that only provide baked illumination or global HDR maps, Lumera-2K provides ground-truth annotations for individual, countable light sources, making parametric lighting a measurable target for the first time.
Key Insights and Ideas
- Decoupling Objects and Lights is Essential: The authors discovered that joint training fails because object tokens are far more frequent than light tokens. In a unified model, the high-frequency box data completely drowns out the rarer light signals. Separating them into distinct VLM streams drastically improved light parsing.
- Structured Tokens over Continuous Fields: Representing complex physical attributes (like light intensity and box dimensions) as discrete text tokens within a VLM schema allows the system to effectively leverage the spatial reasoning capabilities of modern LLMs.
- The Necessity of Constrained Agency: Unconstrained LLM agents often make chaotic changes when trying to fix a scene (e.g., altering a wall's geometry to fix a shadow error). Restricting the agent to a strict field whitelist and separating the geometry and lighting stages keeps the refinement stable and effective.
Key Results
- 3D Box Parsing: Lumera-Box achieved the strongest overall performance against competitive baselines (including DetAny3D, SpatialLM, and WildDet3D) on a sanitized benchmark, scoring a merged mAP of 0.1141, a bounding box IoU (IoU-B) of 0.2472, and an F-score of 0.2762.
- Parametric Light Parsing: Lumera-Light achieved a near-perfect non-empty scene recall of 0.998, meaning it almost never misses the presence of light sources.
- Individual Light Localization: While localizing individual lights remains difficult, at a 0.5-meter threshold, Lumera achieved an F1 score of 0.209, a median position error of 0.261 meters, and a Pearson correlation of 0.628 for light intensity ranking.
Limitations and Future Work
- Heavy-Tailed Asset Distributions: Because the dataset features a massive variety of complex game assets across thousands of projects, strict 3D box overlap metrics remain far from saturated.
- Symmetry and Yaw Errors: The model struggles with yaw estimation for symmetrical objects (e.g., a square table where 0 and 90 degrees look identical).
- Out-of-Frustum Lighting: The pipeline currently struggles to parse near-camera or off-screen lights whose illumination is visible in the scene but whose physical fixtures are out of the camera's field of view.
- Engine Generalization: The data pipeline is currently tailored to Unreal Engine 5 and Blender conventions; extending this to Unity or Godot remains an open engineering challenge.
Summary updated