Introduction
Mixed Reality (MR) aims to seamlessly integrate virtual objects and lighting into physical environments. Achieving realistic interactions, such as a virtual lamp casting shadows on a physical sofa, requires precise 3D scene understanding. However, performing this in real-time on mobile hardware presents a significant computational challenge. Hybrelighter introduces a hybrid framework that overcomes these limitations, delivering high-quality relighting at up to 100 FPS on mobile devices.
The Challenge: Noisy Meshes and Mobile Constraints
To render realistic lighting, MR systems typically rely on LiDAR-generated 3D meshes. On mobile devices, these meshes are often low-fidelity, noisy, and jagged. Direct rendering on such geometry leads to broken visual artifacts. Existing methods generally fall into two categories: 3D-based methods, which require high-quality meshes beyond mobile capabilities, and 2D filter-based methods, which lack the spatial awareness needed for complex occlusions. Hybrelighter bridges this gap by using "messy" 3D meshes for global light placement while employing 2D deep learning to refine the details.
The Hybrelighter Framework
The core of Hybrelighter is a three-stage pipeline designed for efficiency and visual fidelity. First, a Mesh-aware Filtering stage renders a basic lighting layer using the available low-quality mesh. While this provides the general spatial context for shadows and light, it contains significant geometric artifacts.
The second stage, Guided Anisotropic Diffusion, "heals" these artifacts. A lightweight U-Net with a MobileNetV4 backbone extracts high-frequency features (edges and boundaries) from the sharp RGB camera feed. These features guide the diffusion process, ensuring light values spread within object boundaries but do not bleed across them. To maintain high performance, a Cascaded Strategy performs diffusion at low resolutions before progressively upscaling. Finally, a separate Shadow Pass ensures soft, realistic shadows are preserved.
Key Results and Performance
Hybrelighter demonstrates exceptional performance, achieving 100 FPS on high-end GPUs and a stable 50 FPS on an iPhone 16 Pro. In benchmarks focusing on mesh error correction, the system improved perceptual similarity (LPIPS) by 18% compared to standard mesh-based rendering. The cascaded diffusion approach proved to be approximately 9x faster than standard diffusion methods without sacrificing visual quality.
Conclusion
By leveraging the high-resolution RGB feed as a "ground truth" for boundaries, Hybrelighter successfully compensates for the limitations of mobile LiDAR sensors. This hybrid approach allows for realistic, real-time relighting that was previously impossible on mobile hardware. Future work will focus on better distinguishing between scanning artifacts and actual scene shadows to further enhance fidelity in dynamic environments.
Summary updated