Pixel-aware deep function-mixture network for spectral super-resolution

Lei Zhang

TL;DR

This paper introduces a novel "pixel-aware deep function-mixture network" for spectral super-resolution (SSR). It addresses the limitation of existing methods by enabling the network to adaptively determine the optimal receptive field size and mapping function for each individual pixel, leading to significantly improved reconstruction accuracy.

What Problem Does This Paper Address?

Hyperspectral imaging captures detailed spectral information across many narrow bands, making it valuable for applications like remote sensing and medical imaging. However, acquiring high-resolution hyperspectral images (HSIs) is challenging and expensive. Spectral Super-Resolution (SSR) aims to generate a high-spectral-resolution HSI from a lower-spectral-resolution RGB image. Current deep learning approaches for SSR typically use fixed-size receptive fields and a single, universal mapping function for all pixels. This is problematic because different pixels in a scene, due to variations in material, texture, or lighting, often require different contextual information and distinct mapping strategies for accurate spectral reconstruction. The paper tackles this by proposing a method that can learn pixel-specific receptive fields and mapping functions.

Methods

The core innovation is the Function-Mixture (FM) block. Each FM block comprises:

  • Basis Functions: Multiple parallel sub-networks, each with a different receptive field size. These sub-networks learn distinct ways to map RGB context to spectral information.
  • Mixing Function: A dedicated subnet that learns to generate pixel-wise weights. These weights are then used to linearly combine the outputs of the basis functions.

This "mixing" mechanism allows the network to dynamically select and combine information from different receptive fields for each pixel, effectively learning a pixel-dependent mapping function. The authors stack multiple FM blocks to increase the network's capacity for learning complex pixel-wise mappings. Additionally, they incorporate intermediate feature fusion using skip connections to enhance feature reuse and boost performance. The overall architecture is a global residual network built upon these FM blocks.

Datasets

The proposed method was evaluated on three benchmark hyperspectral image datasets:

  • NTIRE2018: A benchmark dataset specifically for SSR, containing 255 paired HSI and RGB images.
  • CAVE: A dataset with 32 HSIs of indoor objects.
  • Harvard: A dataset with 50 HSIs.

For CAVE and Harvard datasets, RGB images were generated from HSIs using the spectral response function of a Nikon D700 camera.

Key Insights and Ideas

  • Pixel-Aware Adaptation is Crucial: The paper strongly advocates that treating all pixels uniformly is a bottleneck in SSR. The ability to adapt receptive field size and mapping function on a per-pixel basis is key to achieving superior results.
  • Function Mixture as a Flexible Mapping: The FM block's function-mixture approach provides a computationally efficient way to achieve pixel-wise adaptation without resorting to complex hypernetworks. It leverages the idea that complex functions can be approximated by a weighted sum of simpler basis functions.
  • Receptive Field Size is Not One-Size-Fits-All: Different scene elements and spatial locations inherently require different amounts and types of contextual information for accurate spectral reconstruction. The FM block directly addresses this by allowing adaptive receptive field selection.
  • Intermediate Feature Fusion Enhances Performance: Reusing features from earlier layers through skip connections is a well-established technique, and the authors demonstrate its effectiveness in their SSR context, further improving the model's ability to capture fine details.

Key Results

The proposed "pixel-aware deep function-mixture network" achieved state-of-the-art performance across the evaluated datasets.

  • Quantitative Improvements: The paper reports significant reductions in RMSE and increases in PSNR compared to existing methods, including bilinear interpolation, sparse recovery methods, and other deep learning approaches like Aitor, HSCNN+, DCNN, and MCNet. For instance, on the NTIRE2018 dataset, the proposed method achieved a PSNR improvement of up to 3.19db over HSCNN+.
  • Qualitative Improvements: Visual results (Figures 4, 5, 6) show that the proposed method produces super-resolved images with more details and fewer reconstruction errors. Recovered spectra (Figure 7) closely match the ground truth.
  • Ablation Studies: The paper demonstrates that both the pixel-wise mixture mechanism and intermediate feature fusion are critical components for the network's success. The number of basis functions and FM blocks also positively impacts performance, with a trade-off between accuracy and computational complexity.

Limitations and Future Work

  • Linear Mixture: The current FM block uses a linear mixture of basis functions. The authors suggest that exploring non-linear mixtures could potentially lead to even better performance.
  • Generalization to Other Tasks: The core idea of pixel-wise modeling through function mixture is promising and could be generalized to other pixel-level tasks such as semantic segmentation and colorization.
  • Computational Complexity: While the method offers significant improvements, increasing the number of basis functions or FM blocks can lead to higher computational costs. Future work might focus on optimizing this trade-off for real-time applications or resource-constrained environments.

Summary updated

Abstract

Spectral super-resolution (SSR) aims at generating a hyperspectral image (HSI) from a given RGB image. Recently, a promising direction is to learn a complicated mapping function from the RGB image to the HSI counterpart using a deep convolutional neural network. This essentially involves mapping the RGB context within a size-specific receptive field centered at each pixel to its spectrum in the HSI. The focus thereon is to appropriately determine the receptive field size and establish the mapping function from RGB context to the corresponding spectrum. Due to their differences in category or spatial position, pixels in HSIs often require different-sized receptive fields and distinct mapping functions. However, few efforts have been invested to explicitly exploit this prior. To address this problem, we propose a pixel-aware deep function-mixture network for SSR, which is composed of a new class of modules, termed function-mixture (FM) blocks. Each FM block is equipped with some basis functions, i.e., parallel subnets of different-sized receptive fields. Besides, it incorporates an extra subnet as a mixing function to generate pixel-wise weights, and then linearly mixes the outputs of all basis functions with those generated weights. This enables us to pixel-wisely determine the receptive field size and the mapping function. Moreover, we stack several such FM blocks to further increase the flexibility of the network in learning the pixel-wise mapping. To encourage feature reuse, intermediate features generated by the FM blocks are fused in late stage, which proves to be effective for boosting the SSR performance. Experimental results on three benchmark HSI datasets demonstrate the superiority of the proposed method.
BibTeX
@inproceedings{zhang2020pixel,
  author = {Lei Zhang},
  title = {{Pixel-aware deep function-mixture network for spectral super-resolution}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i07.6978},
}
Figure