Hybrelighter: Combining Deep Anisotropic Diffusion and Scene Reconstruction for On-device Real-time Relighting in Mixed Reality

Hanwen Zhao, John Akers, Baback Elmieh, Ira Kemelmacher-ShlizermanarXiv 2025 · DOI · arXiv

Introduction

Mixed Reality (MR) aims to seamlessly integrate virtual objects and lighting into physical environments. Achieving realistic interactions, such as a virtual lamp casting shadows on a physical sofa, requires precise 3D scene understanding. However, performing this in real-time on mobile hardware presents a significant computational challenge. Hybrelighter introduces a hybrid framework that overcomes these limitations, delivering high-quality relighting at up to 100 FPS on mobile devices.

The Challenge: Noisy Meshes and Mobile Constraints

To render realistic lighting, MR systems typically rely on LiDAR-generated 3D meshes. On mobile devices, these meshes are often low-fidelity, noisy, and jagged. Direct rendering on such geometry leads to broken visual artifacts. Existing methods generally fall into two categories: 3D-based methods, which require high-quality meshes beyond mobile capabilities, and 2D filter-based methods, which lack the spatial awareness needed for complex occlusions. Hybrelighter bridges this gap by using "messy" 3D meshes for global light placement while employing 2D deep learning to refine the details.

The Hybrelighter Framework

The core of Hybrelighter is a three-stage pipeline designed for efficiency and visual fidelity. First, a Mesh-aware Filtering stage renders a basic lighting layer using the available low-quality mesh. While this provides the general spatial context for shadows and light, it contains significant geometric artifacts.

The Hybrelighter pipeline combines high-resolution RGB data with a lighting overlay derived from 3D scene reconstruction to produce the final relighting result.

The second stage, Guided Anisotropic Diffusion, "heals" these artifacts. A lightweight U-Net with a MobileNetV4 backbone extracts high-frequency features (edges and boundaries) from the sharp RGB camera feed. These features guide the diffusion process, ensuring light values spread within object boundaries but do not bleed across them. To maintain high performance, a Cascaded Strategy performs diffusion at low resolutions before progressively upscaling. Finally, a separate Shadow Pass ensures soft, realistic shadows are preserved.

Key Results and Performance

Hybrelighter demonstrates exceptional performance, achieving 100 FPS on high-end GPUs and a stable 50 FPS on an iPhone 16 Pro. In benchmarks focusing on mesh error correction, the system improved perceptual similarity (LPIPS) by 18% compared to standard mesh-based rendering. The cascaded diffusion approach proved to be approximately 9x faster than standard diffusion methods without sacrificing visual quality.

Comparison of the original scene, the intermediate mesh-based lighting, and the final high-fidelity relighting results under various lighting conditions.

Conclusion

By leveraging the high-resolution RGB feed as a "ground truth" for boundaries, Hybrelighter successfully compensates for the limitations of mobile LiDAR sensors. This hybrid approach allows for realistic, real-time relighting that was previously impossible on mobile hardware. Future work will focus on better distinguishing between scanning artifacts and actual scene shadows to further enhance fidelity in dynamic environments.

Summary updated

Abstract

Mixed Reality scene relighting, where virtual changes to lighting conditions realistically interact with physical objects, producing authentic illumination and shadows, can be used in a variety of applications. One such application in real estate could be visualizing a room at different times of day and placing virtual light fixtures. Existing deep learning-based relighting techniques typically exceed the real-time performance capabilities of current MR devices. On the other hand, scene understanding methods, such as on-device scene reconstruction, often yield inaccurate results due to scanning limitations, in turn affecting relighting quality. Finally, simpler 2D image filter-based approaches cannot represent complex geometry and shadows. We introduce a novel method to integrate image segmentation, with lighting propagation via anisotropic diffusion on top of basic scene understanding, and the computational simplicity of filter-based techniques. Our approach corrects on-device scanning inaccuracies, delivering visually appealing and accurate relighting effects in real-time on edge devices, achieving speeds as high as 100 fps. We show a direct comparison between our method and the industry standard, and present a practical demonstration of our method in the aforementioned real estate example.
BibTeX
@misc{zhao2025hybrelighter,
  author = {Hanwen Zhao and John Akers and Baback Elmieh and Ira Kemelmacher-Shlizerman},
  title = {{Hybrelighter: Combining Deep Anisotropic Diffusion and Scene Reconstruction for On-device Real-time Relighting in Mixed Reality}},
  year = {2025},
  eprint = {2508.14930},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.14930},
}
Figure