Abstract
Spectral Super-Resolution (SSR) aims to recover high-resolution hyperspectral images (HSIs) from standard RGB inputs. Traditional deep learning methods for SSR often employ fixed convolutional kernels, applying a uniform mapping across all image regions regardless of local semantic content. This paper presents the Pixel-Aware Deep Function-Mixture Network, which introduces a novel Function-Mixture (FM) block. By adaptively mixing multiple basis functions with pixel-specific weights, the network dynamically adjusts its receptive field and mapping function for every pixel. Experimental results on NTIRE 2018, CAVE, and Harvard benchmarks demonstrate that this approach achieves state-of-the-art performance in both reconstruction accuracy and spectral fidelity.
Introduction
Hyperspectral images (HSIs) provide a dense "spectral fingerprint" of materials by capturing hundreds of narrow spectral bands, far exceeding the three broad channels (Red, Green, Blue) of standard cameras. While invaluable for tasks like material identification and remote sensing, HSI hardware is often prohibitively expensive or forced to sacrifice spatial resolution for spectral detail. Spectral Super-Resolution (SSR) addresses this by computationally reconstructing HSIs from standard RGB images.
A fundamental challenge in SSR is that different materials require varying levels of spatial context to resolve their spectral signatures. For instance, a pixel in a uniform sky may require minimal context, whereas a pixel in a complex, textured bush requires a larger receptive field to be correctly identified. Most existing models use a fixed receptive field for every pixel, leading to sub-optimal reconstructions in heterogeneous scenes.
Proposed Method: Pixel-Aware Architecture
To overcome the limitations of fixed-mapping networks, the authors propose a network architecture centered around the Function-Mixture (FM) block. The overall pipeline, shown in the figure below, begins with a bicubic upsampling of the RGB input, followed by a series of FM blocks and an intermediate feature fusion stage.
The Function-Mixture (FM) Block
The core innovation is the FM block, which enables "pixel-aware" processing. Instead of a single convolutional path, the block employs a "mixture of experts" approach:
- Basis Functions: Several parallel sub-networks use different-sized convolutional filters (e.g., 3x3, 7x7, and 11x11). This allows the model to capture multi-scale spatial information simultaneously.
- Mixing Function: A separate sub-network generates a set of spatial weight maps (one for each basis function). These weights act as a pixel-wise selector, determining the contribution of each basis function to the final output at that specific location.
- Pixel-Wise Linear Mixing: The final output for each pixel is a weighted sum of the outputs from all basis functions, effectively synthesizing an optimal, custom mapping for every point in the image.
Additionally, the network utilizes Intermediate Feature Fusion. By aggregating features from all previous blocks via skip connections, the model ensures that the final reconstruction incorporates both fine-grained low-level textures and high-level semantic information.
Experimental Results
The model was evaluated on three major HSI benchmarks: NTIRE 2018 (255 images), CAVE (32 images), and Harvard (50 images), all covering the 400nm–700nm range.
Performance Metrics
The proposed method consistently outperformed existing state-of-the-art (SOTA) models, including the NTIRE 2018 challenge winner, HSCNN+:
- Accuracy: On the NTIRE dataset, the model achieved an RMSE of 1.03, a significant improvement over HSCNN+ (1.55) and standard DCNN baselines (1.23).
- Signal Quality: It reached a PSNR of 49.29 dB on NTIRE, approximately 4dB higher than the previous SOTA.
- Semantic Awareness: Visualization of the mixing weights revealed that the network automatically learns to assign different weights to different semantic regions (e.g., roads vs. trees) without explicit supervision, confirming the effectiveness of the pixel-aware strategy.
Conclusion
The Pixel-Aware Deep Function-Mixture Network represents a significant step forward in Spectral Super-Resolution. By replacing static convolutions with a dynamic mixture of basis functions, the model achieves unprecedented flexibility and accuracy. While the parallel branches introduce some computational overhead, the gains in reconstruction quality and the model's ability to generalize across diverse datasets make it a powerful tool for high-fidelity spectral imaging. Future work may explore the model's robustness to real-world sensor noise and its application to other pixel-to-pixel tasks like depth estimation or image colorization.
Summary updated