Pixel-Aware Deep Function-Mixture Network for Spectral Super-Resolution

Lei Zhang, Zhiqiang Lang, Peng Wang, Wei Wei, Shengcai Liao, Ling Shao, Yanning Zhang

Abstract

Spectral Super-Resolution (SSR) aims to recover high-resolution hyperspectral images (HSIs) from standard RGB inputs. Traditional deep learning methods for SSR often employ fixed convolutional kernels, applying a uniform mapping across all image regions regardless of local semantic content. This paper presents the Pixel-Aware Deep Function-Mixture Network, which introduces a novel Function-Mixture (FM) block. By adaptively mixing multiple basis functions with pixel-specific weights, the network dynamically adjusts its receptive field and mapping function for every pixel. Experimental results on NTIRE 2018, CAVE, and Harvard benchmarks demonstrate that this approach achieves state-of-the-art performance in both reconstruction accuracy and spectral fidelity.

Introduction

Hyperspectral images (HSIs) provide a dense "spectral fingerprint" of materials by capturing hundreds of narrow spectral bands, far exceeding the three broad channels (Red, Green, Blue) of standard cameras. While invaluable for tasks like material identification and remote sensing, HSI hardware is often prohibitively expensive or forced to sacrifice spatial resolution for spectral detail. Spectral Super-Resolution (SSR) addresses this by computationally reconstructing HSIs from standard RGB images.

A fundamental challenge in SSR is that different materials require varying levels of spatial context to resolve their spectral signatures. For instance, a pixel in a uniform sky may require minimal context, whereas a pixel in a complex, textured bush requires a larger receptive field to be correctly identified. Most existing models use a fixed receptive field for every pixel, leading to sub-optimal reconstructions in heterogeneous scenes.

Proposed Method: Pixel-Aware Architecture

To overcome the limitations of fixed-mapping networks, the authors propose a network architecture centered around the Function-Mixture (FM) block. The overall pipeline, shown in the figure below, begins with a bicubic upsampling of the RGB input, followed by a series of FM blocks and an intermediate feature fusion stage.

Overview of the Pixel-Aware Deep Function-Mixture Network architecture.

The Function-Mixture (FM) Block

The core innovation is the FM block, which enables "pixel-aware" processing. Instead of a single convolutional path, the block employs a "mixture of experts" approach:

  1. Basis Functions: Several parallel sub-networks use different-sized convolutional filters (e.g., 3x3, 7x7, and 11x11). This allows the model to capture multi-scale spatial information simultaneously.
  2. Mixing Function: A separate sub-network generates a set of spatial weight maps (one for each basis function). These weights act as a pixel-wise selector, determining the contribution of each basis function to the final output at that specific location.
  3. Pixel-Wise Linear Mixing: The final output for each pixel is a weighted sum of the outputs from all basis functions, effectively synthesizing an optimal, custom mapping for every point in the image.
Detailed structure of the Function-Mixture Block (FMB) showing pixel-wise weight generation and basis function mixing.

Additionally, the network utilizes Intermediate Feature Fusion. By aggregating features from all previous blocks via skip connections, the model ensures that the final reconstruction incorporates both fine-grained low-level textures and high-level semantic information.

Experimental Results

The model was evaluated on three major HSI benchmarks: NTIRE 2018 (255 images), CAVE (32 images), and Harvard (50 images), all covering the 400nm–700nm range.

Performance Metrics

The proposed method consistently outperformed existing state-of-the-art (SOTA) models, including the NTIRE 2018 challenge winner, HSCNN+:

  • Accuracy: On the NTIRE dataset, the model achieved an RMSE of 1.03, a significant improvement over HSCNN+ (1.55) and standard DCNN baselines (1.23).
  • Signal Quality: It reached a PSNR of 49.29 dB on NTIRE, approximately 4dB higher than the previous SOTA.
  • Semantic Awareness: Visualization of the mixing weights revealed that the network automatically learns to assign different weights to different semantic regions (e.g., roads vs. trees) without explicit supervision, confirming the effectiveness of the pixel-aware strategy.

Conclusion

The Pixel-Aware Deep Function-Mixture Network represents a significant step forward in Spectral Super-Resolution. By replacing static convolutions with a dynamic mixture of basis functions, the model achieves unprecedented flexibility and accuracy. While the parallel branches introduce some computational overhead, the gains in reconstruction quality and the model's ability to generalize across diverse datasets make it a powerful tool for high-fidelity spectral imaging. Future work may explore the model's robustness to real-world sensor noise and its application to other pixel-to-pixel tasks like depth estimation or image colorization.

Summary updated

Abstract

Spectral super-resolution (SSR) aims at generating a hyperspectral image (HSI) from a given RGB image. Recently, a promising direction is to learn a complicated mapping function from the RGB image to the HSI counterpart using a deep convolutional neural network. This essentially involves mapping the RGB context within a size-specific receptive field centered at each pixel to its spectrum in the HSI. The focus thereon is to appropriately determine the receptive field size and establish the mapping function from RGB context to the corresponding spectrum. Due to their differences in category or spatial position, pixels in HSIs often require different-sized receptive fields and distinct mapping functions. However, few efforts have been invested to explicitly exploit this prior.To address this problem, we propose a pixel-aware deep function-mixture network for SSR, which is composed of a new class of modules, termed function-mixture (FM) blocks. Each FM block is equipped with some basis functions, i.e., parallel subnets of different-sized receptive fields. Besides, it incorporates an extra subnet as a mixing function to generate pixel-wise weights, and then linearly mixes the outputs of all basis functions with those generated weights. This enables us to pixel-wisely determine the receptive field size and the mapping function. Moreover, we stack several such FM blocks to further increase the flexibility of the network in learning the pixel-wise mapping. To encourage feature reuse, intermediate features generated by the FM blocks are fused in late stage, which proves to be effective for boosting the SSR performance. Experimental results on three benchmark HSI datasets demonstrate the superiority of the proposed method.
BibTeX
@inproceedings{zhang2020pixel,
  author = {Lei Zhang and Zhiqiang Lang and Peng Wang and Wei Wei and Shengcai Liao and Ling Shao and Yanning Zhang},
  title = {{Pixel-Aware Deep Function-Mixture Network for Spectral Super-Resolution}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i07.6978},
}
Figure