TL;DR
This paper introduces a novel two-stage approach for generating aesthetically pleasing relit face images from a single input image and a target light direction. By leveraging multi-level attention mechanisms and a new synthetic dataset, the model achieves state-of-the-art results, even adapting to real-world images through unsupervised domain adaptation.
What Problem Does This Paper Address?
Face relighting, the task of altering the lighting on a face in an image, has numerous applications in areas like photo editing and facial recognition. A key challenge is not just accurately rendering shadows but doing so in a way that looks natural and "photo-realistic." Existing methods often struggle with this, producing images with unnatural shadows or lacking fine-grained details. This can stem from limitations in training data (e.g., assuming Lambertian surfaces) or the inherent difficulty of capturing complex illumination effects. Furthermore, generalizing models trained on synthetic data to real-world images is a significant hurdle due to domain shift.
Methods
The proposed solution employs a two-stage training pipeline:
- Stage 1: Residual Convolutional Autoencoder with Multi-Level Attention:
- A lighting network encodes the target light position and intensity into a feature embedding.
- This embedding is combined with features extracted from the input RGB image (and its luminance channel) by a residual convolutional autoencoder.
- Crucially, Multi DConv Head Attention (MDHA) modules are integrated at skip connections between the encoder and decoder. These modules operate at higher resolutions, allowing the network to capture fine-grained facial and shadow details that are often lost in lower-resolution feature maps. This is a key innovation for achieving aesthetic results.
- The training loss for this stage is a combination of global and local image reconstruction losses (Smooth L1 and DSSIM), a light loss, and a VGG perceptual loss.
- Stage 2: Unsupervised Domain Adaptation with GANs:
- To bridge the gap between synthetic training data and real-world images, the Stage 1 model is used as a generator in a Generative Adversarial Network (GAN) framework.
- A discriminator is trained to distinguish between real (ground truth) and fake (generated) relit images.
- The training objective includes Smooth L1 loss, DSSIM loss, and a relativistic adversarial loss. This stage aims to improve the perceptual quality and adapt the model to real image distributions without requiring paired real-world training data.
Datasets
- Synthetic OLAT Lighting Rig Dataset: The authors created a new dataset using a synthetic One-Light-At-a-Time (OLAT) lighting rig in Blender. This dataset features:
- 8 synthetic 3D human models.
- 3,000 input-relit image pairs per model.
- Dense sampling of light source positions and intensities within a unit hemisphere.
- Inclusion of specular and diffuse reflections, leading to more photo-realistic ground truth compared to previous datasets.
- Variations in 3D model orientation and camera position to enhance generalization.
- This dataset differs from the DPR dataset by having more light source positions per image and by accounting for specular reflections.
- Evaluation Datasets:
- Real Human Test Dataset: A custom test set created from the synthetic data, featuring out-of-training light source positions.
- Multi-pie Dataset: A standard dataset for face analysis, used for evaluating performance on real images.
- Celeb-FFHQ Dataset: Used for qualitative comparisons.
Key Insights and Ideas
- Attention at Higher Resolutions is Key for Aesthetics: The core insight is that fine-grained facial and shadow details, crucial for aesthetic relighting, are best captured by attention mechanisms operating at higher spatial resolutions. This directly addresses the "not aesthetic/photo-realistic" problem of prior methods.
- Synthetic Data with Richer Illumination is Sufficient: By carefully constructing a synthetic dataset with diverse lighting conditions and realistic reflections, the authors demonstrate that it's possible to train a model that generalizes well to real images, even without direct real-world supervision in the initial stage.
- Unsupervised Domain Adaptation is Effective: The GAN-based approach effectively bridges the synthetic-to-real gap, improving perceptual quality and enabling the model to handle real-world variations without expensive paired data collection.
- Implicit Shadow Learning: The architecture implicitly learns the relationship between illumination and shadows, rather than relying on explicit shadow masks or geometric cues, leading to more natural results.
Key Results
- Quantitative Performance: The proposed model (especially Stage 2) outperforms prior state-of-the-art methods (Zhou et al. [47], Hou et al. [9, 8]) on real image datasets (Real Human Test Dataset and Multi-pie) across multiple metrics, including MSE (RGB and Hue), DSSIM, and LPIPS. For instance, on the Real Human test dataset, the Stage 2 model achieves a significantly lower MSE (0.0049 vs. 0.0152 for Hou et al. [8]) and DSSIM (0.0336 vs. 0.0787 for Hou et al. [8]).
- Qualitative Performance: Visual results show that the model generates more aesthetic relit images with accurate and soft shadows, closely resembling ground truth. It also effectively corrects skin tone issues observed in Stage 1. The model demonstrates strong generalization to out-of-training light source positions.
- Efficiency: The model is relatively lightweight, with approximately 4.45 million parameters, and achieves superior performance despite being trained on a significantly smaller synthetic dataset (21,000 images) compared to some prior works.
Limitations and Future Work
- Skin Tone Issues on Multi-pie: The authors acknowledge some skin tone (color) differences on the Multi-pie dataset. This is partly attributed to the input images having a blueish hue and being captured in low-light conditions, which the current model does not explicitly model (i.e., light color is not considered).
- Segmentation Mask Dependency: The current approach relies on a segmentation mask to isolate the face region.
- Future Directions:
- Automatically understanding and segmenting the face region without an explicit mask.
- Jointly modeling ambient and source lighting, as well as light color, to create more immersive relit images akin to studio portraits.
Summary updated