TL;DR
This paper presents a novel method for "inverse rendering" – recovering the 3D shape, surface properties (albedo), and lighting of outdoor scenes from a single, unconstrained image. They achieve this by training a deep neural network using self-supervision derived from a differentiable renderer and, crucially, by leveraging offline multiview stereo (MVS) reconstructions for additional supervision.
What Problem Does This Paper Address?
Inverse rendering aims to decompose an image into its constituent physical properties: shape, reflectance (how light bounces off surfaces), and illumination (the light sources in the scene). This is a fundamental problem in computer vision with applications ranging from augmented reality to image editing. However, obtaining accurate ground truth data for real-world, uncontrolled scenes is extremely difficult, making traditional supervised learning approaches infeasible. Existing methods often rely on synthetic data (which may not generalize well) or are limited to specific object classes (like faces) or controlled environments. This paper tackles the challenging problem of performing scene-level inverse rendering for arbitrary outdoor scenes using only single, unconstrained images, without requiring ground truth.
Methods
The core of the proposed method is a fully convolutional neural network that takes a single RGB image as input and outputs several intermediate maps:
- Normal Map: Represents the surface orientation at each pixel.
- Diffuse Albedo Map: Represents the intrinsic color of the surface, independent of lighting.
- Shadow Map: A scalar map that accounts for shadowing effects.
These predicted maps are then used to infer Spherical Harmonic Lighting Coefficients, which model the scene's illumination.
The key innovation lies in the self-supervision strategy. Since ground truth is unavailable, they use a differentiable renderer to synthesize an image from the network's predicted shape, albedo, and lighting. An "appearance loss" is then computed between this rendered image and the original input image.
To overcome the inherent ambiguity of single-image inverse rendering, they introduce crucial additional supervision from offline Multiview Stereo (MVS) reconstructions. This MVS data provides:
- Coarse Normal Map Supervision: The depth maps from MVS are used to generate initial normal maps, which provide direct supervision during training.
- Photometric Invariant Consistency: By cross-projecting between overlapping views in the MVS data, they enforce consistency of photometric invariants (properties that should remain the same under varying illumination) using a Siamese network training approach. This helps to disentangle illumination from reflectance.
They also incorporate a statistical prior on natural illumination learned from a large dataset of HDR environment maps.
Datasets
- Training: The primary training dataset is MegaDepth [28], a large collection of uncontrolled outdoor images with associated depth maps. They also leverage BigTime time-lapse sequences [43] and NYU dataset [76] for specific training stages and fine-tuning.
- Evaluation:
- MegaDepth [28]: Used for evaluating overall inverse rendering performance.
- BigTime [43]: Used for evaluating albedo consistency under varying illumination.
- IIW (Intrinsic Images in the Wild) [73]: An indoor benchmark used for evaluating intrinsic image decomposition.
- DIODE [74]: A benchmark for geometry estimation, used to evaluate normal map prediction on outdoor scenes.
- [75]: A benchmark dataset for illumination estimation, consisting of multi-illumination and multi-view images of a monument.
Key Insights and Ideas
- MVS as a powerful supervisory signal for inverse rendering: The paper demonstrates that even offline MVS reconstructions, which are not perfect and can have holes, provide invaluable geometric and correspondence information that significantly improves the learning of inverse rendering from single images.
- Self-supervision with a differentiable renderer is feasible but insufficient alone: While a differentiable renderer enables learning from appearance loss, it's not enough to resolve the inherent ambiguities. Additional constraints are critical.
- Disentangling shadow from albedo is crucial: Explicitly predicting a shadow map allows the network to avoid "baking" shadows into the albedo, leading to more accurate surface property estimation.
- Leveraging photometric invariants across views is key to disentangling illumination and reflectance: By enforcing consistency of these invariants between MVS-derived views, the network learns to separate what is intrinsic to the surface from what is due to the lighting.
- A statistical prior on natural illumination helps constrain the problem: Not all lighting combinations are natural; learning these statistical regularities guides the network towards more plausible solutions.
Key Results
- The proposed method achieves state-of-the-art or competitive results on various benchmarks for inverse rendering, normal estimation, and intrinsic image decomposition.
- Quantitative results on MegaDepth show improved reflectance (albedo) and normal estimation compared to previous methods. For instance, their method achieves an MSE of 0.0093 for reflectance and a median angular error of 31.2 degrees for normals.
- Qualitative results demonstrate the ability to recover plausible albedo, normal, and shading maps from single images, even in complex outdoor scenes with varying illumination. They show successful separation of shadows from albedo.
- The method is shown to generalize to indoor scenes (IIW benchmark) and perform well on normal prediction for outdoor scenes (DIODE).
Limitations and Future Work
- Failure Cases: The method struggles with dark plaques that are primarily explained by shadowing rather than dark albedo, and performance degrades in regions where MVS supervision is lacking (e.g., ground planes, dynamic objects).
- Color Ambiguity: There remains a color ambiguity between illumination and albedo, although the intensity distribution is plausible.
- Reflective Surfaces: The current model does not explicitly handle specular reflections, and MVS supervision may not correctly deal with reflective surfaces.
- Future Work: The authors suggest several directions:
- Relaxing modeling assumptions (e.g., more general reflectance models, estimating global illumination effects).
- Combining with dedicated depth prediction networks.
- Incorporating losses from intrinsic image decomposition networks.
- Extending the lighting prior to better handle indoor scenes.
- Making reflectance and illumination models partially learnable.
Summary updated