Abstract
This paper reviews the second challenge on spectral reconstruction from RGB images, i.e., the recovery of whole-scene hyperspectral (HS) information from a 3-channel RGB image. As in the previous challenge, two tracks were provided: (i) a "Clean" track where HS images are estimated from noise-free RGBs, the RGB images are themselves calculated numerically using the ground-truth HS images and supplied spectral sensitivity functions (ii) a "Real World" track, simulating capture by an uncalibrated and unknown camera, where the HS images are recovered from noisy JPEG-compressed RGB images. A new, larger-than-ever, natural hyperspectral image data set is presented, containing a total of 510 HS images. The Clean and Real World tracks had 103 and 78 registered participants respectively, with 14 teams competing in the final testing phase. A description of the proposed methods, alongside their challenge scores and an extensive evaluation of top performing methods is also provided. They gauge the state-of-the-art in spectral reconstruction from an RGB image.
Introduction
Spectral reconstruction is the mathematically "underdetermined" task of recovering high-resolution hyperspectral data—often dozens of narrow spectral bands—from standard 3-channel RGB images. While hyperspectral imaging (HSI) provides a unique "fingerprint" for materials, the hardware is traditionally expensive and slow. The NTIRE 2020 challenge sought to advance software-based recovery methods, utilizing the new ARAD HS dataset to test models under both ideal ("Clean") and degraded ("Real World") conditions.
Methods and Architectural Trends
The 2020 challenge saw a complete transition toward deep learning architectures. To address the ill-posed nature of recovering ~31 spectral values from 3 inputs, teams employed several sophisticated strategies:
- Attention Mechanisms: The winning method, IPIC_SSR, utilized an Adaptive Weighted Attention Network (AWAN) to dynamically weigh different channels and spatial regions, focusing on the most informative features.
- Hierarchical Regression: OrangeCat employed a 4-level hierarchical network (HRNet) to process features at multiple scales, maintaining spatial resolution while capturing deep semantic features.
- Pixel-Aware Networks: Methods like MDISL-lab used "function-mixture" blocks to determine receptive field sizes pixel-by-pixel, acknowledging that different materials require varying amounts of spatial context.
The ARAD HS Dataset
A primary contribution of this challenge is the ARAD HS Dataset, which significantly expands the resources available for this field. Captured using a Specim IQ mobile hyperspectral camera, the dataset contains 510 high-quality images—nearly double the size of previous benchmarks. The images were resampled to 31 spectral bands covering the visual range (400nm to 700nm) at 10nm increments.
Key Insights and Evaluation
Beyond raw error metrics, the challenge provided critical insights into how these models function:
- Spatial Context Dependency: Through "spatial shuffling" tests, organizers found that performance plummeted when image patches were randomized. This proves that models are not merely performing a color-to-spectrum mapping; they are using object recognition and texture cues to "guess" material spectra.
- Exposure Robustness: While 2020 models are more robust than their predecessors, they still struggle with "exposure invariance," showing performance degradation when image brightness is significantly altered.
- Physical Consistency: A gap remains in physical consistency; when reconstructed spectra are projected back into RGB space, they often fail to perfectly match the original input, indicating errors that might be visible in color-critical applications.
Results
The IPIC_SSR team achieved the top score in the Clean track with a Mean Relative Absolute Error (MRAE) of 0.0301. In the Real World track, OrangeCat led with an MRAE of 0.0620. The transition from clean to real-world data roughly doubled the error rates, highlighting the significant impact of JPEG artifacts and sensor noise on spectral recovery. Computational efficiency also varied wildly, with processing times ranging from 0.034 seconds to over 16 seconds per image.
Conclusion and Future Work
While deep learning has achieved remarkably low error rates, challenges remain in generalizing to out-of-scope lighting conditions and achieving real-time performance on mobile devices. Future research is expected to focus on lightweight "edge" models, improved generalization across diverse environments, and the extension of these techniques to video streams.
Summary updated