Towards a Perceptual Evaluation Framework for Lighting Estimation

Justine Giroux, Mohammad Reza Karimi Dastjerdi, Yannick Hold-Geoffroy, Javier Vazquez-Corral, Jean-François LalondeCVPR 2024 · DOI

Abstract

Lighting estimation is a cornerstone of realistic augmented reality and visual effects, yet its evaluation has historically relied on mathematical image quality metrics that may not reflect human judgment. This paper presents a rigorous investigation into the alignment between automated metrics and human perception. Through a large-scale user study involving 49 observers across accuracy and plausibility tasks, we demonstrate that standard metrics often fail to predict human preferences, especially for glossy materials. We propose a novel learned metric that combines multiple image quality assessments to bridge this gap, achieving significantly higher correlation with human perception.

Introduction

In computer vision, lighting estimation algorithms aim to recover scene illumination from a single image. This capability is essential for seamlessly integrating virtual objects into real-world photographs. Traditionally, the field has measured progress by comparing renders produced with estimated lighting against those using ground-truth lighting, employing standard Image Quality Assessment (IQA) metrics such as PSNR, SSIM, or LPIPS.

However, a fundamental question remains: do these mathematical metrics correlate with what humans perceive as realistic? In practical applications, users lack access to ground truth; they simply require that the inserted object looks believable within its context. This research investigates whether current automated metrics are leading the community toward suboptimal solutions by failing to capture the nuances of human visual perception.

Methodology: A Psychophysical Approach

To quantify human perception of lighting, the authors conducted a controlled user study with 49 observers. The study utilized two distinct Two-Alternative Forced Choice (2AFC) tasks:

  1. Accuracy Task: Observers compared two estimated renders against a ground-truth reference to determine which best matched the original lighting.
  2. Plausibility Task: Observers viewed two renders composited into a background without a reference, selecting the one that appeared more realistic.

The experiment spanned 50 HDR panoramas (25 indoor, 25 outdoor) and tested both diffuse (matte) and glossy (shiny) materials. Several state-of-the-art lighting estimation methods were evaluated against these human benchmarks. Following data collection, 15 standard IQA metrics were tested for their predictive power. Finally, the authors developed a learned metric using a Support Vector Regressor (SVR) to aggregate these 15 metrics into a single, perceptually-aligned score.

Key Insights: The Gap Between Math and Perception

The study revealed several critical discrepancies between automated metrics and human judgment:

  • Task-Specific Cognition: Humans evaluate lighting accuracy and plausibility using different visual cues. Matching a reference is a different cognitive process than judging contextual realism.
  • Material Sensitivity: While parametric light source models suffice for diffuse objects, they fail for glossy objects. Glossy materials require high-frequency HDR textures to produce believable reflections, a nuance often missed by simple metrics.
  • The Failure of Standard Metrics: Most common IQA metrics (e.g., SSIM, FLIP) performed no better than random chance when predicting human preferences for glossy objects or plausibility tasks. They only showed reliable agreement in the simplest case: accuracy matching for diffuse materials.

Key Results

The proposed learned metric represents a significant advancement in lighting evaluation. By training on human preference data, the metric achieved agreement scores as high as 0.889 for glossy plausibility tasks, whereas individual standard metrics often hovered near the 0.5 baseline (equivalent to a coin toss).

Comparison of human preferences, standard metrics, and the proposed learned metric across different scenes. The proposed metric consistently aligns with human judgment where standard metrics fail.

Furthermore, the learned metric demonstrated strong generalization capabilities, accurately predicting human preferences for lighting estimation methods not included in its training set, such as those based on Stable Diffusion outpainting.

Conclusion and Future Work

This research highlights a significant evaluation gap in the field of lighting estimation. Standard metrics are insufficient for capturing the complexities of human visual realism, particularly for specular reflections and contextual plausibility. The proposed learned metric offers a more robust benchmark for the community. Future work should continue to explore how specific algorithmic choices—such as lighting representation—impact human perception, moving beyond simple image-level comparisons toward a deeper understanding of visual believability.

Summary updated

BibTeX
@inproceedings{giroux2024towards,
  author = {Justine Giroux and Mohammad Reza Karimi Dastjerdi and Yannick Hold-Geoffroy and Javier Vazquez-Corral and Jean-François Lalonde},
  title = {{Towards a Perceptual Evaluation Framework for Lighting Estimation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.00422},
}
Figure