NTIRE 2020 Challenge on Spectral Reconstruction from an RGB Image

Boaz Arad, Radu Timofte, Ohad Ben-Shahar, Yi-Tun Lin, Graham D. Finlayson, Shai Givati, Jiaojiao Li, Chaoxiong Wu, Rui Song, Yunsong Li, Fei Liu, Zhiqiang Lang, Wei Wei, Lei Zhang, Jiangtao Nie, Yuzhi Zhao, Lai-Man Po, Qiong Yan, Wei Liu, Tingyu Lin, Youngjung Kim, Changyeop Shin, Kyeongha Rho, Sungho Kim, Zhiyu Zhu, Junhui Hou, He Sun, Jinchang Ren, Zhenyu Fang, Yijun Yan, Hao Peng, Xiaomei Chen, Jie Zhao, Tarek Stiebel, Simon Koppers, Dorit Merhof, Honey Gupta, Kaushik Mitra, Biebele Joslyn Fubara, Mohamed H. Sedky, Dave Dyke, Atmadeep Banerjee, Akash Palrecha, Sabarinathan sabarinathan, K. Uma, D. Synthiya Vinothini, B. Sathya Bama, S. M. Md Mansoor Roomi

Abstract

This paper reviews the second challenge on spectral reconstruction from RGB images, i.e., the recovery of whole-scene hyperspectral (HS) information from a 3-channel RGB image. As in the previous challenge, two tracks were provided: (i) a "Clean" track where HS images are estimated from noise-free RGBs, the RGB images are themselves calculated numerically using the ground-truth HS images and supplied spectral sensitivity functions (ii) a "Real World" track, simulating capture by an uncalibrated and unknown camera, where the HS images are recovered from noisy JPEG-compressed RGB images. A new, larger-than-ever, natural hyperspectral image data set is presented, containing a total of 510 HS images. The Clean and Real World tracks had 103 and 78 registered participants respectively, with 14 teams competing in the final testing phase. A description of the proposed methods, alongside their challenge scores and an extensive evaluation of top performing methods is also provided. They gauge the state-of-the-art in spectral reconstruction from an RGB image.

Introduction

Spectral reconstruction is the mathematically "underdetermined" task of recovering high-resolution hyperspectral data—often dozens of narrow spectral bands—from standard 3-channel RGB images. While hyperspectral imaging (HSI) provides a unique "fingerprint" for materials, the hardware is traditionally expensive and slow. The NTIRE 2020 challenge sought to advance software-based recovery methods, utilizing the new ARAD HS dataset to test models under both ideal ("Clean") and degraded ("Real World") conditions.

The 2020 challenge saw a complete transition toward deep learning architectures. To address the ill-posed nature of recovering ~31 spectral values from 3 inputs, teams employed several sophisticated strategies:

  • Attention Mechanisms: The winning method, IPIC_SSR, utilized an Adaptive Weighted Attention Network (AWAN) to dynamically weigh different channels and spatial regions, focusing on the most informative features.
  • Hierarchical Regression: OrangeCat employed a 4-level hierarchical network (HRNet) to process features at multiple scales, maintaining spatial resolution while capturing deep semantic features.
  • Pixel-Aware Networks: Methods like MDISL-lab used "function-mixture" blocks to determine receptive field sizes pixel-by-pixel, acknowledging that different materials require varying amounts of spatial context.
Common architectural components used in deep learning models for spectral reconstruction.

The ARAD HS Dataset

A primary contribution of this challenge is the ARAD HS Dataset, which significantly expands the resources available for this field. Captured using a Specim IQ mobile hyperspectral camera, the dataset contains 510 high-quality images—nearly double the size of previous benchmarks. The images were resampled to 31 spectral bands covering the visual range (400nm to 700nm) at 10nm increments.

Sample images from the ARAD HS dataset illustrating the diversity of natural scenes.

Key Insights and Evaluation

Beyond raw error metrics, the challenge provided critical insights into how these models function:

  • Spatial Context Dependency: Through "spatial shuffling" tests, organizers found that performance plummeted when image patches were randomized. This proves that models are not merely performing a color-to-spectrum mapping; they are using object recognition and texture cues to "guess" material spectra.
  • Exposure Robustness: While 2020 models are more robust than their predecessors, they still struggle with "exposure invariance," showing performance degradation when image brightness is significantly altered.
  • Physical Consistency: A gap remains in physical consistency; when reconstructed spectra are projected back into RGB space, they often fail to perfectly match the original input, indicating errors that might be visible in color-critical applications.

Results

The IPIC_SSR team achieved the top score in the Clean track with a Mean Relative Absolute Error (MRAE) of 0.0301. In the Real World track, OrangeCat led with an MRAE of 0.0620. The transition from clean to real-world data roughly doubled the error rates, highlighting the significant impact of JPEG artifacts and sensor noise on spectral recovery. Computational efficiency also varied wildly, with processing times ranging from 0.034 seconds to over 16 seconds per image.

Conclusion and Future Work

While deep learning has achieved remarkably low error rates, challenges remain in generalizing to out-of-scope lighting conditions and achieving real-time performance on mobile devices. Future research is expected to focus on lightweight "edge" models, improved generalization across diverse environments, and the extension of these techniques to video streams.

Summary updated

Abstract

This paper reviews the second challenge on spectral reconstruction from RGB images, i.e., the recovery of whole-scene hyperspectral (HS) information from a 3-channel RGB image. As in the previous challenge, two tracks were provided: (i) a "Clean" track where HS images are estimated from noise-free RGBs, the RGB images are themselves calculated numerically using the ground-truth HS images and supplied spectral sensitivity functions (ii) a "Real World" track, simulating capture by an uncalibrated and unknown camera, where the HS images are recovered from noisy JPEG-compressed RGB images. A new, larger-than-ever, natural hyperspectral image data set is presented, containing a total of 510 HS images. The Clean and Real World tracks had 103 and 78 registered participants respectively, with 14 teams competing in the final testing phase. A description of the proposed methods, alongside their challenge scores and an extensive evaluation of top performing methods is also provided. They gauge the state-of-the-art in spectral reconstruction from an RGB image.
BibTeX
@inproceedings{arad2020ntire,
  author = {Boaz Arad and Radu Timofte and Ohad Ben-Shahar and Yi-Tun Lin and Graham D. Finlayson and Shai Givati and Jiaojiao Li and Chaoxiong Wu and Rui Song and Yunsong Li and Fei Liu and Zhiqiang Lang and Wei Wei and Lei Zhang and Jiangtao Nie and Yuzhi Zhao and Lai-Man Po and Qiong Yan and Wei Liu and Tingyu Lin and Youngjung Kim and Changyeop Shin and Kyeongha Rho and Sungho Kim and Zhiyu Zhu and Junhui Hou and He Sun and Jinchang Ren and Zhenyu Fang and Yijun Yan and Hao Peng and Xiaomei Chen and Jie Zhao and Tarek Stiebel and Simon Koppers and Dorit Merhof and Honey Gupta and Kaushik Mitra and Biebele Joslyn Fubara and Mohamed H. Sedky and Dave Dyke and Atmadeep Banerjee and Akash Palrecha and Sabarinathan sabarinathan and K. Uma and D. Synthiya Vinothini and B. Sathya Bama and S. M. Md Mansoor Roomi},
  title = {{NTIRE 2020 Challenge on Spectral Reconstruction from an RGB Image}},
  booktitle = {Workshops @ CVPR},
  year = {2020},
  doi = {10.1109/cvprw50498.2020.00231},
}
Figure