Light stage super-resolution: continuous high-frequency relighting

Tiancheng Sun, Zexiang Xu, Xiuming Zhang, Sean Ryan Fanello, Christoph Rhemann, Paul E. Debevec, Yun-Ta Tsai, Jonathan T. Barron, Ravi Ramamoorthi

Abstract

Light stages are essential for capturing human reflectance fields but are limited by discrete LED sampling, leading to "ghosting" artifacts during high-frequency relighting. This paper presents a deep learning framework that super-resolves angular resolution using an alias-free pooling mechanism and specialized training. The model synthesizes photorealistic faces under arbitrary light directions, outperforming traditional interpolation methods even with significantly fewer physical light sources.

Introduction

Capturing the reflectance field of a human face typically involves a light stage—a spherical rig of LEDs. By capturing a subject under one light at a time (OLAT), researchers can simulate any lighting environment. However, physical constraints limit the density of these rigs. Sparse sampling causes issues when simulating "virtual" light positions between physical LEDs. Traditional linear blending works for soft, low-frequency lighting but fails for sharp, high-frequency lighting, resulting in "ghosting" artifacts—multiple faint shadows and specular highlights instead of a single, moving light source.

Methodology

The proposed method treats relighting as a feature-space interpolation problem rather than a pixel-space one. The architecture consists of parallel encoding, alias-free pooling, and feature-based decoding.

Active Set Selection and Pooling

For a given query light direction, the system selects the $k$ nearest physical lights. These images are processed through a shared convolutional encoder. To ensure temporal stability and avoid flickering as the active set changes, the authors introduced Alias-Free Pooling. This mechanism uses a weighted averaging scheme where weights transition to zero as a light enters or leaves the active set.

Figure 6: The training strategy uses irregular sampling to force the model to generalize beyond the fixed hexagonal grid of the physical light stage.

Irregularity Training

A key challenge is the regularity of physical light stages. To prevent the network from overfitting to a specific hexagonal grid, the authors employed "irregularity training." By randomly sampling subsets of neighbors during training, the model learns to handle the arbitrary distributions it will encounter at test time, as illustrated in the training vs. testing comparison in Figure 6.

Results

The model demonstrates a significant improvement in visual quality, particularly in the rendering of sharp specularities and cast shadows. By moving the blending process into the latent space of a neural network, the model learns to "move" highlights rather than simply fading them in and out.

Figure 3: Comparison of relighting results. The proposed method eliminates ghosting artifacts and produces sharp, realistic shadows and highlights.

Performance and Generalization

Quantitative metrics (RMSE, DSSIM, and LPIPS) confirm that the learning-based approach outperforms previous state-of-the-art methods. Notably, the model trained on a sparse set of 100 lights produced superior results to linear blending on a dense set of 302 lights. This suggests that computational super-resolution can compensate for lower-cost hardware setups. The system also generalizes effectively to unseen subjects and diverse skin tones, accurately capturing complex light transport like sub-surface scattering.

Conclusion and Future Work

While the method provides a robust solution for angular super-resolution, it faces challenges with high-resolution (4K) output due to GPU memory constraints. Future work may address these limitations and explore the application of this architecture to non-human subjects with more complex material properties, such as fur or refractive glass.

Summary updated

Abstract

The light stage has been widely used in computer graphics for the past two decades, primarily to enable the relighting of human faces. By capturing the appearance of the human subject under different light sources, one obtains the light transport matrix of that subject, which enables image-based relighting in novel environments. However, due to the finite number of lights in the stage, the light transport matrix only represents a sparse sampling on the entire sphere. As a consequence, relighting the subject with a point light or a directional source that does not coincide exactly with one of the lights in the stage requires interpolation and resampling the images corresponding to nearby lights, and this leads to ghosting shadows, aliased specularities, and other artifacts. To ameliorate these artifacts and produce better results under arbitrary high-frequency lighting, this paper proposes a learning-based solution for the "super-resolution" of scans of human faces taken from a light stage. Given an arbitrary "query" light direction, our method aggregates the captured images corresponding to neighboring lights in the stage, and uses a neural network to synthesize a rendering of the face that appears to be illuminated by a "virtual" light source at the query location. This neural network must circumvent the inherent aliasing and regularity of the light stage data that was used for training, which we accomplish through the use of regularized traditional interpolation methods within our network. Our learned model is able to produce renderings for arbitrary light directions that exhibit realistic shadows and specular highlights, and is able to generalize across a wide variety of subjects. Our super-resolution approach enables more accurate renderings of human subjects under detailed environment maps, or the construction of simpler light stages that contain fewer light sources while still yielding comparable quality renderings as light stages with more densely sampled lights.
BibTeX
@article{sun2020light,
  author = {Tiancheng Sun and Zexiang Xu and Xiuming Zhang and Sean Ryan Fanello and Christoph Rhemann and Paul E. Debevec and Yun-Ta Tsai and Jonathan T. Barron and Ravi Ramamoorthi},
  title = {{Light stage super-resolution: continuous high-frequency relighting}},
  journal = {ACM Trans. Graphics},
  year = {2020},
  doi = {10.1145/3414685.3417821},
}
Figure