Environment Maps Editing using Inverse Rendering and Adversarial Implicit Functions

Antonio D'Orazio, Davide Sforza, Fabio Pellacini, Iacopo MasiarXiv 2024 · DOI · OpenReview

Abstract

This paper presents a novel "lazy-learning" approach for editing High Dynamic Range (HDR) environment maps through 2D brush strokes. By representing the environment map as a robust neural implicit function (R-SIREN) and optimizing it via a differentiable renderer, we enable intuitive lighting modifications—such as shifting shadows or introducing reflections—while preserving the visual coherence of the background. Our method avoids the artifacts common in direct pixel manipulation by optimizing in the weight space of the neural representation.

Introduction

Environment maps are essential for realistic lighting in computer graphics but remain challenging to edit. Traditional methods often require manual HDR pixel manipulation, which frequently introduces noise and artifacts. Automating this process via inverse rendering faces three primary hurdles: the sparsity of light sources, the extreme dynamic range of pixel values, and the difficulty of maintaining visual consistency. This work addresses these challenges by parameterizing the environment map as a neural network and optimizing its weights to match user-defined targets.

Methodology

The proposed pipeline shifts the optimization target from individual pixels to the weights of a neural network.

HDR SIREN and R-SIREN

We utilize a Sinusoidal Representation Network (SIREN) to represent the environment map. To accommodate HDR values, the network is trained in a normalized logarithmic space. We employ a dual-loss strategy: one loss ensures structural integrity in log-space, while another focuses on high-intensity peaks in HDR-space. To prevent the optimization from producing "unnatural" maps (e.g., static noise), we introduce Adversarial Weight Perturbation (R-SIREN). This technique forces the model to find smooth regions in the weight space, acting as a powerful implicit prior for natural-looking structures.

Overview of the SIREN HDR optimization pipeline, showing the flow from coordinates to the final rendered loss.

Inverse Differentiable Rendering

The system integrates the Mitsuba 3 differentiable renderer. Gradients derived from a target image—where a user has painted desired changes like a new shadow—are backpropagated to update the R-SIREN weights. This process is further stabilized by a regularization suite including luminance stability, $L_1$ sparsity for light sources, and the DISTS perceptual metric.

Results

The R-SIREN approach was evaluated using the PolyHaven dataset, specifically focusing on "PureSky" maps.

Quantitative and Qualitative Performance

Quantitatively, our method outperforms standard pixel-based optimization. For instance, in the "Horse" scene, R-SIREN HDR achieved a PSNR of ~29.7 dB, significantly higher than the ~18.3 dB achieved by pixel optimization. Qualitatively, the method handles complex edits such as moving shadows and adding reflections with high fidelity. Unlike baseline methods that often result in "salt-and-pepper" noise in the sky, our approach maintains visual coherence.

Qualitative results demonstrating the system's ability to move shadows and add reflections while maintaining background integrity.

Conclusion

We have demonstrated a lightweight, "on-the-fly" method for environment map editing that leverages the implicit priors of neural networks. While the system currently relies on reflections and may hallucinate light sources in unobserved regions, it provides a robust foundation for intuitive lighting control. Future work will explore extending this framework to 3D model manipulation and incorporating more diverse training data to further refine the local latent space of the neural functions.

Summary updated

Abstract

Editing High Dynamic Range (HDR) environment maps using an inverse differentiable rendering architecture is a complex inverse problem due to the sparsity of relevant pixels and the challenges in balancing light sources and background. The pixels illuminating the objects are a small fraction of the total image, leading to noise and convergence issues when the optimization directly involves pixel values. HDR images, with pixel values beyond the typical Standard Dynamic Range (SDR), pose additional challenges. Higher learning rates corrupt the background during optimization, while lower learning rates fail to manipulate light sources. Our work introduces a novel method for editing HDR environment maps using a differentiable rendering, addressing sparsity and variance between values. Instead of introducing strong priors that extract the relevant HDR pixels and separate the light sources, or using tricks such as optimizing the HDR image in the log space, we propose to model the optimized environment map with a new variant of implicit neural representations able to handle HDR images. The neural representation is trained with adversarial perturbations over the weights to ensure smooth changes in the output when it receives gradients from the inverse rendering. In this way, we obtain novel and cheap environment maps without relying on latent spaces of expensive generative models, maintaining the original visual consistency. Experimental results demonstrate the method's effectiveness in reconstructing the desired lighting effects while preserving the fidelity of the map and reflections on objects in the scene. Our approach can pave the way to interesting tasks, such as estimating a new environment map given a rendering with novel light sources, maintaining the initial perceptual features, and enabling brush stroke-based editing of existing environment maps.
BibTeX
@misc{dorazio2024environment,
  author = {Antonio D'Orazio and Davide Sforza and Fabio Pellacini and Iacopo Masi},
  title = {{Environment Maps Editing using Inverse Rendering and Adversarial Implicit Functions}},
  year = {2024},
  eprint = {2410.18622},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.18622},
}
Figure