TL;DR
This paper introduces a novel "gated mixture-of-experts" neural network for efficiently capturing the complex reflectance properties of materials. By intelligently routing input data to specialized network components, it achieves higher quality results with fewer captured images compared to previous state-of-the-art methods.
What Problem Does This Paper Address?
Digitizing the visual appearance of physical materials, known as reflectance capture, is crucial for applications like visual effects, e-commerce, and gaming. The goal is to represent how a material reflects light from different angles and under various lighting conditions, often using a complex model called a spatially-varying bidirectional reflectance distribution function (SVBRDF). Traditional methods for capturing this data are extremely time-consuming and storage-intensive, requiring thousands of photographs. While recent advancements use illumination multiplexing and neural networks to speed up acquisition, they often employ a single, unified network. This paper addresses the inefficiency of unified networks, which can struggle with the diversity of material appearances, leading to a trade-off between generality and quality.
Methods
The core innovation is a "deep gated mixture-of-experts" neural network. This architecture consists of:
- Gating Module: This module takes photometric measurements (images captured under specific lighting) as input and predicts a probability distribution over a set of specialized "expert" decoders. It learns to identify which expert is best suited to process the given input.
- Specialized Decoders: Instead of one large network, there are multiple smaller, specialized decoders. Each decoder is trained to reconstruct reflectance for a specific subset of material appearances. This specialization allows for higher quality reconstruction by trading generality for focused expertise.
- Latent-Transform Module: To manage the increased number of decoders, a common, pre-trained module is used by each decoder. This module transforms the decoder's output into a latent representation, effectively reducing the burden on individual decoders.
- Joint Optimization: The system also jointly optimizes the illumination conditions used during capture with the neural network's parameters. This allows the acquisition process itself to be tailored for better reconstruction.
Datasets
The paper does not explicitly mention specific named datasets for training or evaluation. Instead, it describes generating synthetic training data by creating 200 million virtual lumitexels (collections of photometric measurements for a single point) based on the GGX BRDF model and the device's calibration data. For evaluation, they use 5 sets of challenging near-planar physical samples with diverse appearances.
Key Insights and Ideas
- Specialization Trumps Generalization: The key insight is that by dividing the problem into smaller, specialized tasks (each handled by a dedicated decoder), the network can achieve higher quality results than a single, general-purpose network. This is analogous to having experts in different fields rather than one general practitioner.
- Adaptive Routing for Efficiency: The gating module is crucial for efficiency. It intelligently selects the most appropriate expert for each input, avoiding unnecessary computation and improving the overall processing speed.
- Joint Optimization of Capture and Reconstruction: Optimizing both the lighting patterns and the neural network simultaneously leads to a more effective and efficient capture pipeline.
- Latent Space for Scalability: Using a pre-trained latent-transform module is a clever way to scale the mixture-of-experts approach. It allows for more decoders without a proportional increase in computational complexity for each decoder.
Key Results
- Improved Efficiency: The proposed method achieves higher reconstruction quality with the same number of input images compared to the state-of-the-art LDAE method.
- Reduced Input Requirements: For comparable quality, the number of input images (lighting patterns) can be reduced by approximately 1/3. For example, they achieve similar results to LDAE using 12 lighting patterns compared to LDAE's 32.
- High-Quality Reconstruction: The method successfully reconstructs complex anisotropic reflectance, separating diffuse and specular components, and producing high-quality normal maps.
- Runtime: Training takes approximately 72 hours for 1 million iterations, with pre-training of the autoencoder taking 60 hours. Runtime for decoding 1 million lumitexel pairs is 5 minutes, and subsequent parameter fitting takes 1.5 hours.
Limitations and Future Work
- Unexpected Outputs: Similar to existing neural acquisition methods, the system can produce unexpected outputs for physical lumitexels that significantly deviate from the training data.
- Global Illumination: The current framework does not account for global illumination effects.
- Future Directions: The authors suggest exploring spectral domain multiplexing for further efficiency gains, investigating load-balancing techniques for mixture-of-experts, and extending the framework to handle more general appearance phenomena like subsurface scattering.
Summary updated