Multilspectral snapshot demosaicing via non-convex matrix completion

Giancarlo A. Antonucci, Simon Vary, David Humphreys, Robert A. Lamb, Jonathan Piper, Jared Tanner

Abstract: Snapshot mosaic multispectral imagery acquires an undersampled data cube by acquiring a single spectral measurement per spatial pixel. Sensors which acquire $p$ frequencies, therefore, suffer from severe $1/p$ undersampling of the full data cube. We show that the missing entries can be accurately imputed using non-convex techniques from sparse approximation and matrix completion initialised with traditional demosaicing algorithms. In particular, we observe the peak signal-to-noise ratio can typically be improved by 2 to 5 dB over current state-of-the-art methods when simulating a $p=16$ mosaic sensor measuring both high and low altitude urban and rural scenes as well as ground-based scenes.

The Challenge of Snapshot Multispectral Imaging

Multispectral imaging—capturing an image across many different frequencies of light—is a powerful tool for applications ranging from landmine detection to precision agriculture. However, traditional sensors are often bulky or slow. Snapshot mosaic sensors solve this by tiling different spectral filters over a single 2D sensor, similar to a standard camera's Bayer filter but with many more spectral bands.

The primary drawback is extreme undersampling. In a 16-band sensor, each pixel records only one specific frequency, meaning 93.75% of the data is missing. The process of filling in these gaps, known as demosaicing, is notoriously difficult. Traditional methods like bilinear interpolation often result in blurry edges or severe spectral artifacts, such as "rainbowing" or color fringing on high-contrast edges.

Figure 1: Visual artifacts (rainbowing) typical of traditional demosaicing in snapshot mosaic sensors.

Methodology: From Local Interpolation to Global Optimization

To recover this lost data, this research moves beyond "local" interpolation to exploit the underlying mathematical structure of natural scenes. The authors compare three primary approaches:

  1. Direct Interpolation (Baselines): Methods like Weighted Bilinear (WB), Spectral Difference (SD), and Intensity Difference (ID) use neighboring pixels to estimate missing values.
  2. Sparse Approximation (Compressed Sensing): This assumes the image can be represented compactly in a different domain, using 2D Wavelets for spatial detail and Discrete Cosine Transform (DCT) for spectral smoothness.
  3. Matrix Completion (The Core Contribution): By "unfolding" the 3D data cube into a large matrix, the researchers treat demosaicing as a matrix completion problem. Because different spectral bands in a natural scene are highly correlated, this matrix is naturally "low-rank."

The key innovation lies in using non-convex solvers—specifically Conjugate Gradient Iterative Hard Thresholding (CGIHT) and Alternating Steepest Descent (ASD). Crucially, these solvers are "warm-started" using the output of traditional interpolation (like Spectral Difference), which prevents the algorithms from converging to poor local minima.

Key Results and Performance

The study evaluated these algorithms across diverse environments, including high-altitude airborne data (AVIRIS), low-altitude agricultural imagery, and ground-based datasets. All data was downsampled to simulate a 16-band, 4x4 mosaic sensor.

Quantitative Performance

The proposed matrix completion methods (CGIHT and ASD) consistently outperformed state-of-the-art baselines. As shown in the performance metrics, these methods provide a significant boost in Peak Signal-to-Noise Ratio (PSNR), typically ranging from 2 dB to 5 dB across the visible and near-infrared spectrum.

Figure 2: PSNR performance across wavelengths (nm) for various demosaicing methods. CGIHT and ASD consistently outperform traditional interpolation and sparse approximation.

Visual Fidelity

Beyond the numbers, the matrix completion approach preserves sharp edges and fine details much better than wavelets or interpolation. While wavelets can sometimes introduce "yellowing" or contrast issues, matrix completion maintains the structural integrity of the scene, making the reconstructions often visually indistinguishable from the original, fully-sampled data.

Conclusion

By treating multispectral demosaicing as a low-rank matrix completion problem and utilizing non-convex optimization with intelligent initialization, researchers can overcome the extreme undersampling inherent in snapshot sensors. This approach offers a robust path forward for high-speed, high-fidelity multispectral imaging in real-world environments.

Summary updated

Abstract

Snapshot mosaic multispectral imagery acquires an undersampled data cube by acquiring a single spectral measurement per spatial pixel. Sensors which acquire $p$ frequencies, therefore, suffer from severe $1/p$ undersampling of the full data cube. We show that the missing entries can be accurately imputed using non-convex techniques from sparse approximation and matrix completion initialised with traditional demosaicing algorithms. In particular, we observe the peak signal-to-noise ratio can typically be improved by 2 to 5 dB over current state-of-the-art methods when simulating a $p=16$ mosaic sensor measuring both high and low altitude urban and rural scenes as well as ground-based scenes.
BibTeX
@inproceedings{antonucci2019multilspectral,
  author = {Giancarlo A. Antonucci and Simon Vary and David Humphreys and Robert A. Lamb and Jonathan Piper and Jared Tanner},
  title = {{Multilspectral snapshot demosaicing via non-convex matrix completion}},
  booktitle = {DSW},
  year = {2019},
  url = {https://openreview.net/forum?id=t21iU1cIsK},
}
Figure