TL;DR
This paper introduces "symmetric photography," a novel technique to efficiently capture high-resolution 8D reflectance fields. By exploiting the inherent data-sparseness and symmetry of light transport, their method significantly reduces acquisition time and data storage requirements compared to previous approaches, enabling more realistic image-based rendering and relighting.
What Problem Does This Paper Address?
Capturing a full 8D reflectance field, which describes how light interacts with a scene from all viewpoints and under all lighting conditions, is crucial for realistic computer graphics. However, the sheer size of this data (potentially billions of entries) makes direct measurement computationally intractable, requiring days or even months of capture time. Existing methods often capture lower-dimensional slices of the reflectance field, limiting their flexibility in relighting or changing viewpoints. This paper aims to bridge this gap by developing an efficient method to acquire the complete 8D reflectance field.
Methods
The core of the paper lies in two key observations about light transport:
- Data-sparseness: While the transport matrix can be large, its information content is often compressible. This means many sub-blocks can be represented by low-rank approximations, even if they contain high-frequency details.
- Symmetry: Due to Helmholtz reciprocity, the light transport from direction A to direction B is the same as from B to A. This symmetry allows for simultaneous measurements of incoming and outgoing light transport.
Based on these, the authors propose "symmetric photography." They introduce a hierarchical tensor data structure (specifically, a "sedectree" for 4D tensors) to represent the reflectance field. This structure adaptively subdivides the transport matrix, allowing for local low-rank approximations. Their hierarchical acquisition algorithm iteratively refines this representation. At each step, it uses a coaxial projector-camera setup to capture images and determines which sub-blocks can be represented by rank-1 approximations. By exploiting symmetry, they can measure corresponding row and column sums simultaneously, enabling efficient rank-1 factorization. A graph coloring approach is used to schedule projector patterns efficiently, minimizing capture time.
Datasets
The paper evaluates their method on several scenes:
- Figure 1: A glass full of gummy bears, used to demonstrate sub-surface scattering and relighting.
- Figure 2: Four simple scenes consisting of planes (diffuse textured, two diffuse white planes, diffuse white and textured plane, two diffuse textured planes) to illustrate the structure of the transport matrix and the hierarchical decomposition.
- Figure 5 & 6: More complex scenes demonstrating the capture of full 8D reflectance fields, allowing for arbitrary viewpoint and lighting changes.
The paper does not introduce new benchmark datasets but rather uses these example scenes to showcase the capabilities of their acquisition technique.
Key Insights and Ideas
- Data-sparseness is key: The paper moves beyond traditional sparsity (few non-zero elements) to "data-sparseness," where a matrix can have many non-zero elements but still be compactly represented by low-rank approximations. This is crucial for high-dimensional data like reflectance fields.
- Symmetry as a powerful tool: Exploiting Helmholtz reciprocity allows for a significant reduction in the number of measurements needed, as information can be gathered from both illumination and view directions simultaneously.
- Hierarchical tensors for efficient representation: This data structure naturally maps to the adaptive subdivision and low-rank approximation strategy, providing a compact and efficient way to store and access the complex reflectance field data.
- Coaxial setup is ideal: A coaxial projector-camera system simplifies the geometry and enables the direct application of symmetry principles, making the acquisition process more streamlined.
Key Results
- Dramatic speedup: The symmetric photography approach requires approximately 2 orders of magnitude fewer patterns than a brute-force scan to capture reflectance fields. For example, Figure 5 required only 3,368 HDR images over 8 hours, compared to an estimated 233,657 images for a brute-force approach.
- Reduced data size: The hierarchical tensor representation significantly compresses the reflectance field data. For the scene in Figure 5, the compressed size was 1.47 GB, compared to a potential 610 GB for a naive representation.
- High-quality reconstructions: The captured reflectance fields allow for realistic relighting and viewpoint changes, faithfully reconstructing effects like shadows, specularities, and sub-surface scattering, as demonstrated in the figures.
- Effective hierarchical acquisition: The algorithm successfully identifies data-sparse regions and uses them to accelerate the acquisition process, as visualized in Figure 2(d) showing the hierarchy levels.
Limitations and Future Work
- Setup limitations: The use of a beam-splitter in their coaxial setup introduces slight double imaging and can reduce signal-to-noise ratio (SNR). Lens flare from optical components can also cause non-symmetric artifacts (Figure 7).
- 4D tensor simplification: The current implementation uses a 4th-order hierarchical tensor, which flattens two dimensions of the light field. An 8th-order tensor would exploit more of the data's coherency, potentially leading to more efficient acquisition.
- Interpolation of reflectance fields: While techniques exist for interpolating light fields, interpolating full reflectance fields remains an open problem. Future work could explore flow-based techniques for denser sampling.
- Hardware advancements: The authors hope future hardware will offer more symmetric projector/camera combinations, reducing the need for complex calibration and beam-splitters.
- Parallelization: The processing time for the acquisition algorithm is significant. Implementing a parallelized version could further reduce overall acquisition times.
Summary updated