Abstract
SpecGen is a novel neural framework designed to estimate full spectral Bidirectional Reflectance Distribution Functions (BRDFs) from a single RGB image of a material sphere. By leveraging a Spectral-Spatial Tri-plane Aggregation (SSTA) module and an Adaptive Feature Fusion (AFF) mechanism, the model overcomes the extreme scarcity of spectral datasets. It achieves state-of-the-art performance, outperforming existing spectral reconstruction methods by approximately 8 dB in PSNR and enabling high-fidelity rendering across the visible and near-infrared spectrum (425nm to 988nm).
The Challenge of Spectral BRDFs
In computer graphics, accurately modeling how materials reflect light is fundamental to photorealistic rendering. This is captured by the Bidirectional Reflectance Distribution Function (BRDF). While RGB-based BRDF estimation has seen significant progress, spectral BRDFs—which define reflectance as a function of wavelength—remain difficult to acquire. Measuring a spectral BRDF requires high-resolution scanning across a four-dimensional domain (incident/outgoing angles and wavelength), a process that is both tedious and time-consuming.
Consequently, while RGB datasets like MERL contain hundreds of materials, public spectral BRDF datasets (such as RGL) are limited to a few dozen samples. Existing "spectral uplifting" techniques can convert RGB images to spectral images but fail to recover the underlying material properties necessary for rendering under novel lighting or on different geometries. SpecGen addresses this gap by generating the full material "recipe" from a single photograph.
Methodology: Spectral-Spatial Tri-plane Aggregation (SSTA)
The core of SpecGen is the SSTA module, which represents the 4D BRDF function—dependent on three angular variables $(\theta_h, \theta_d, \phi_d)$ and wavelength $(\lambda)$—using a decomposed tri-plane architecture.
Tri-plane Decomposition and Fusion
Inspired by K-Planes, the model utilizes two distinct sets of tri-planes:
- Spatial Tri-planes: These encode the geometric reflectance response across different incident and outgoing angles. These attributes are largely shared between RGB and spectral representations.
- Spectral Tri-planes: These capture the unique reflectance characteristics across the wavelength dimension.
To combine these features, SpecGen introduces the Adaptive Feature Fusion (AFF) module. Unlike standard methods that use simple element-wise multiplication, AFF employs an attention-based mechanism to dynamically weight features. This allows the network to adaptively select the most relevant receptive fields for various material types, significantly boosting reconstruction accuracy.
RGB-Spectral Joint Training
To mitigate data scarcity, SpecGen employs a joint training strategy. Because the spatial tri-planes are shared, the model can be trained simultaneously on limited spectral data and much larger RGB datasets (like MERL). For RGB samples, the spectral dimension is "collapsed" by averaging features, allowing the model to learn complex geometric reflectance patterns from abundant RGB data while specializing in spectral signatures using the RGL dataset.
Key Results and Performance
SpecGen demonstrates a significant leap over existing hyperspectral reconstruction baselines such as HD-Net and MST++. Quantitative evaluations show an average improvement of 8 dB in PSNR.
The model's strengths include:
- Broad Spectrum Coverage: It handles a range from 425nm to 988nm, extending well into the near-infrared, whereas many competitors are limited to the 400–700nm visible range.
- Generalization: SpecGen successfully renders complex shapes, such as the Stanford Bunny, under novel environmental lighting conditions (both indoor and outdoor) not seen during training.
- Efficiency: By using an Implicit Neural Representation (INR), the model remains resolution-independent and memory-efficient.
Conclusion and Future Directions
SpecGen effectively bridges the gap between common RGB imagery and high-fidelity spectral material modeling. By decoupling spatial geometry from spectral signatures, it maximizes the utility of existing datasets. Future work will focus on relaxing the requirement for spherical input geometry and improving the handling of highly specular, mirror-like materials which remain sensitive to input lighting conditions.
Summary updated