Measured Albedo in the Wild: Filling the Gap in Intrinsics Evaluation

Jiaye Wu, Sanjoy Chowdhury, Hariharmano Shanmugaraja, David Jacobs, Roni Sengupta

Abstract

Intrinsic image decomposition—the separation of an image into its reflectance (albedo) and illumination (shading) components—is a fundamental task in computer vision. For years, the field has relied on the Weighted Human Disagreement Rate (WHDR) on the Intrinsic Images in the Wild (IIW) dataset as the primary benchmark. However, this paper identifies a critical "blind spot": the community has over-optimized for this ordinal metric, leading to models that achieve high scores while producing visually distorted albedo with incorrect colors and textures. To address this, we introduce Measured Albedo in the Wild (MAW), a dataset featuring ground-truth albedo measurements, alongside three new metrics that evaluate absolute intensity, color accuracy, and texture preservation.

Introduction: The WHDR Trap

In the standard formulation $I = A \cdot S$, where $I$ is the image, $A$ is albedo, and $S$ is shading, the goal is to recover $A$ and $S$ from a single input. The IIW dataset and its associated WHDR metric evaluate models based on human judgments of relative lightness (e.g., "is point A darker than point B?"). While useful for capturing human perception, WHDR is an ordinal metric that ignores absolute physical values.

Research reveals a "WHDR Paradox": models that excel at relative ranking often produce albedo maps with washed-out intensities, "odd" color tints, or blurry textures. By treating the problem as a ranking task rather than a physical measurement task, the field has inadvertently encouraged models to "flatten" albedo to satisfy relative constraints at the expense of physical realism.

Comparison of albedo estimation results showing the impact of different decomposition approaches on a desk scene.

The MAW Dataset and Methodology

To bridge the gap between perception and physics, the authors developed the Measured Albedo in the Wild (MAW) dataset. MAW contains 888 images across 46 indoor scenes, providing dense ground-truth albedo for 434 unique objects.

The Proxy Object Measurement

To obtain ground truth in uncontrolled environments, the researchers employed a "proxy object" technique. By placing a standard 18% gray card (a surface with a known, calibrated albedo) onto a target surface, they could capture the local shading. By comparing images with and without the card, the true albedo of the underlying surface can be mathematically derived. This method allows for high-quality, dense ground truth in real-world settings without the need for complex laboratory equipment.

A typical indoor scene from the MAW dataset used to evaluate intrinsic decomposition in real-world environments.

A Holistic Evaluation Framework

Beyond the dataset, the paper proposes three complementary metrics to hold models accountable for physical accuracy:

  1. Intensity (si-MSE): A scale-invariant Mean Squared Error that measures how well the absolute lightness of the predicted albedo matches the measured ground truth.
  2. Chromaticity (Color Error): Utilizing the CIEDE2000 standard, this metric isolates perceptual color differences, ensuring the model correctly identifies the "hue" of a surface regardless of lighting.
  3. Texture Error: Using the LPIPS perceptual similarity metric on regions of constant shading, this catches "texture leaking," where surface details mistakenly migrate from the albedo into the shading component.

Key Results and Insights

Benchmarking nine state-of-the-art (SOTA) algorithms revealed that high performance on IIW does not guarantee physical accuracy. For instance, models trained on synthetic data (like CGIntrinsics) often exhibit better color and intensity accuracy than those optimized solely for WHDR, despite lower IIW scores.

Furthermore, the authors demonstrate that finetuning existing models on the MAW dataset yields substantial improvements. On average, finetuned models saw a 36% improvement in intensity accuracy, an 18% improvement in chromaticity, and an 8% improvement in texture preservation. Qualitatively, these models produce significantly more natural-looking albedo maps, eliminating the unnatural color tints (such as pinkish or greenish hues) common in previous outputs.

Conclusion

The introduction of MAW and its associated metrics provides a necessary course correction for intrinsic image decomposition research. By moving beyond ordinal rankings toward physical measurement, the field can develop models that are not only perceptually plausible but also physically accurate. Future work will look to expand this framework to include non-planar surfaces and more complex, non-diffuse lighting models.

Summary updated

Abstract

Intrinsic image decomposition and inverse rendering are long-standing problems in computer vision. To evaluate albedo recovery, most algorithms report their quantitative performance with a mean Weighted Human Disagreement Rate (WHDR) metric on the IIW dataset. However, WHDR focuses only on relative albedo values and often fails to capture overall quality of the albedo. In order to comprehensively evaluate albedo, we collect a new dataset, Measured Albedo in the Wild (MAW), and propose three new metrics that complement WHDR: intensity, chromaticity and texture metrics. We show that existing algorithms often improve WHDR metric but perform poorly on other metrics. We then finetune different algorithms on our MAW dataset to significantly improve the quality of the reconstructed albedo both quantitatively and qualitatively. Since the proposed intensity, chromaticity, and texture metrics and the WHDR are all complementary we further introduce a relative performance measure that captures average performance. By analysing existing algorithms we show that there is significant room for improvement. Our dataset and evaluation metrics will enable researchers to develop algorithms that improve albedo reconstruction.
BibTeX
@inproceedings{wu2023measured,
  author = {Jiaye Wu and Sanjoy Chowdhury and Hariharmano Shanmugaraja and David Jacobs and Roni Sengupta},
  title = {{Measured Albedo in the Wild: Filling the Gap in Intrinsics Evaluation}},
  booktitle = {ICCP},
  year = {2023},
  url = {https://openreview.net/forum?id=UO4Bi0AzzF},
}

Loading related content...

Figure