Light stage super-resolution: continuous high-frequency relighting

Tiancheng Sun, Zexiang Xu, Xiuming Zhang, Sean Ryan Fanello, Christoph Rhemann, Paul E. Debevec, Yun-Ta Tsai, Jonathan T. Barron, Ravi Ramamoorthi

Abstract

Light stages are essential for capturing human reflectance fields but are limited by discrete LED sampling, leading to "ghosting" artifacts during high-frequency relighting. This paper presents a deep learning framework that super-resolves angular resolution using an alias-free pooling mechanism and specialized training. The model synthesizes photorealistic faces under arbitrary light directions, outperforming traditional interpolation methods even with significantly fewer physical light sources.

Introduction

Capturing the reflectance field of a human face typically involves a light stage—a spherical rig of LEDs. By capturing a subject under one light at a time (OLAT), researchers can simulate any lighting environment. However, physical constraints limit the density of these rigs. Sparse sampling causes issues when simulating "virtual" light positions between physical LEDs. Traditional linear blending works for soft, low-frequency lighting but fails for sharp, high-frequency lighting, resulting in "ghosting" artifacts—multiple faint shadows and specular highlights instead of a single, moving light source.

Methodology

The proposed method treats relighting as a feature-space interpolation problem rather than a pixel-space one. The architecture consists of parallel encoding, alias-free pooling, and feature-based decoding.

Active Set Selection and Pooling

For a given query light direction, the system selects the $k$ nearest physical lights. These images are processed through a shared convolutional encoder. To ensure temporal stability and avoid flickering as the active set changes, the authors introduced Alias-Free Pooling. This mechanism uses a weighted averaging scheme where weights transition to zero as a light enters or leaves the active set.

Figure 6: The training strategy uses irregular sampling to force the model to generalize beyond the fixed hexagonal grid of the physical light stage.

Irregularity Training

A key challenge is the regularity of physical light stages. To prevent the network from overfitting to a specific hexagonal grid, the authors employed "irregularity training." By randomly sampling subsets of neighbors during training, the model learns to handle the arbitrary distributions it will encounter at test time, as illustrated in the training vs. testing comparison in Figure 6.

Results

The model demonstrates a significant improvement in visual quality, particularly in the rendering of sharp specularities and cast shadows. By moving the blending process into the latent space of a neural network, the model learns to "move" highlights rather than simply fading them in and out.

Figure 3: Comparison of relighting results. The proposed method eliminates ghosting artifacts and produces sharp, realistic shadows and highlights.

Performance and Generalization

Quantitative metrics (RMSE, DSSIM, and LPIPS) confirm that the learning-based approach outperforms previous state-of-the-art methods. Notably, the model trained on a sparse set of 100 lights produced superior results to linear blending on a dense set of 302 lights. This suggests that computational super-resolution can compensate for lower-cost hardware setups. The system also generalizes effectively to unseen subjects and diverse skin tones, accurately capturing complex light transport like sub-surface scattering.

Conclusion and Future Work

While the method provides a robust solution for angular super-resolution, it faces challenges with high-resolution (4K) output due to GPU memory constraints. Future work may address these limitations and explore the application of this architecture to non-human subjects with more complex material properties, such as fur or refractive glass.

Summary updated

BibTeX
@article{sun2020light,
  author = {Tiancheng Sun and Zexiang Xu and Xiuming Zhang and Sean Ryan Fanello and Christoph Rhemann and Paul E. Debevec and Yun-Ta Tsai and Jonathan T. Barron and Ravi Ramamoorthi},
  title = {{Light stage super-resolution: continuous high-frequency relighting}},
  journal = {ACM Trans. Graphics},
  year = {2020},
  doi = {10.1145/3414685.3417821},
}

Loading related content...

Figure