Abstract
Traditional intrinsic image decomposition focuses on decomposing images into reflectance and shading, leaving surface normals and lighting entangled in shading. In this work, we propose a Global-Local Spherical Harmonics (GLoSH) lighting model to improve the lighting component and jointly predict reflectance and surface normals. The global SH models the holistic lighting while local SH accounts for the spatial variation of lighting. Also, a novel non-negative lighting constraint is proposed to encourage the estimated SH to be physically meaningful. To seamlessly reflect the GLoSH model, we design a coarse-to-fine network structure. Lacking labels for reflectance and lighting, we apply synthetic data for model pre-training and fine-tune the model with real data in a self-supervised way. Compared to state-of-the-art methods, our method recovers all components and achieves consistently better results on three real datasets: IIW, SAW, and NYUv2.
Introduction
Intrinsic image decomposition is the process of "unmixing" an image into its fundamental physical components: reflectance (the true color of the surface) and shading (how light interacts with the surface). A major challenge in computer vision is that shading itself is a complex product of object geometry (surface normals) and the lighting environment. Most existing methods lump these together as a single "shading" layer, leaving geometry and light entangled.
Modeling lighting is particularly difficult because it varies across a scene due to shadows and inter-reflections. Simple global models are too rigid, while per-pixel models often lack the necessary constraints to remain physically plausible. GLoSH addresses this by explicitly modeling lighting as a combination of global and local components.
Methodology
The GLoSH Lighting Model
The authors propose the GLoSH (Global-Local Spherical Harmonics) model, which represents lighting as the sum of two parts:
- Global SH: Captures the holistic lighting of the entire scene.
- Local Residual SH: Captures spatial variations like shadows and highlights.
This provides a "middle ground" that is flexible enough for complex scenes but constrained enough to remain physically grounded. To ensure the light is physically valid, a non-negative constraint is applied by sampling thousands of points on a virtual sphere and penalizing negative intensity values during training.
Coarse-to-Fine Network
The system uses a two-stage CNN architecture. The "Coarse Net" predicts global lighting, reflectance, and normals at a low resolution. The "Fine Net" then predicts local residuals to sharpen these components and add spatial lighting variations, resulting in high-fidelity decompositions.
Evaluation and Results
Because dense ground truth for reflectance and lighting is rare in real-world photos, the model is pre-trained on the synthetic SUNCG dataset and then fine-tuned on real images using self-supervision (reconstruction loss) and sparse human-labeled datasets like IIW (Intrinsic Images in the Wild) and SAW (Shading Annotations in the Wild).
Key Findings
- Reflectance: GLoSH achieved the best performance on the IIW dataset with a Weighted Human Disagreement Rate (WHDR) of 15.2%, outperforming previous state-of-the-art methods.
- Shading: The model produced shading images with significantly better visual contrast and fewer artifacts than specialized shading estimation methods.
- Normals: Jointly estimating light and shape helped the model understand geometry better than estimating shape alone, producing highly competitive results on the NYUv2 dataset.
- Disentanglement: Unlike prior work, GLoSH correctly attributes cast shadows and highlights to the shading/lighting component rather than "bleeding" them into the reflectance (albedo) map.
Conclusion
By explicitly modeling lighting as Global-Local Spherical Harmonics, GLoSH successfully disentangles the complex relationship between light, shape, and color. The combination of physical constraints and a coarse-to-fine architecture allows for a more complete and accurate intrinsic decomposition from a single RGB image than previously possible.
Summary updated