Multispectral Snapshot Demosaicing Via Non-Convex Matrix Completion

Giancarlo A. Antonucci, Simon Vary, David Humphreys, Robert A. Lamb, Jonathan Piper, Jared Tanner

TL;DR

This paper tackles the challenge of reconstructing full multispectral images from severely undersampled "snapshot mosaic" sensors. By combining traditional demosaicing techniques with advanced non-convex matrix completion methods, they achieve significant improvements in image quality, outperforming existing state-of-the-art by 2-5 dB in peak signal-to-noise ratio.

What Problem Does This Paper Address?

Multispectral imaging captures rich spectral information, enabling detailed material analysis across various fields like landmine detection and precision agriculture. However, acquiring full multispectral data cubes is often hindered by large sensor sizes and long acquisition times. "Snapshot mosaic" sensors offer a solution by capturing only one spectral band per pixel in a spatial mosaic. While this dramatically speeds up acquisition and reduces sensor complexity, it leads to severe undersampling (e.g., 1/16th of the full data cube for a 16-band sensor). The core problem is how to accurately reconstruct the missing spectral information from this highly incomplete data.

Methods

The authors propose a two-stage approach. First, they leverage traditional demosaicing algorithms like spatial bilinear interpolation with spectral correlation (SD) and intensity difference (ID) to obtain initial estimates of the missing data. These initial estimates are then used to initialize more sophisticated non-convex matrix completion (MC) and sparse approximation (SA) algorithms.

Specifically, they explore:

  • Traditional Interpolation: SD and ID methods are used as baseline and for initialization.
  • Sparse Approximation (SA): This method assumes the image can be sparsely represented in a transformed domain (e.g., a combination of 2D wavelets for spatial information and Discrete Cosine Transform for spectral information). They solve a minimization problem to reconstruct the data.
  • Matrix Completion (MC): This approach exploits the low-rank structure inherent in multispectral data. They formulate the problem as finding a low-rank matrix that best fits the observed data. They specifically employ non-convex solvers like Conjugate Gradient Iterative Hard Thresholding (CGIHT) and Alternating Steepest Descent (ASD).

The key novelty lies in using these advanced non-convex techniques, initialized by traditional methods, to handle the extreme undersampling characteristic of snapshot mosaic sensors.

Datasets

The paper evaluates their methods on a variety of datasets:

  • High altitude airborne images: AVIRIS and IEEE GRSS Data Fusion contest datasets.
  • Low altitude airborne images: 10 representative radiance images from DSTL Porton Down.
  • Ground-based images: Datasets from Stanford SCIEN, Nascimento, and Foster.

These datasets were processed to simulate the undersampling patterns of a p=16 mosaic sensor.

Key Insights and Ideas

  • Initialization is Crucial: Traditional demosaicing methods, while not optimal on their own, provide essential initial estimates that significantly boost the performance of advanced matrix completion and sparse approximation techniques.
  • Matrix Completion Excels: Non-convex matrix completion methods, particularly when initialized with traditional techniques, consistently outperform both traditional interpolation and sparse approximation methods in reconstructing multispectral data from severe undersampling.
  • Low-Rank Structure is Key: The success of matrix completion suggests that multispectral data, even when undersampled, possesses a strong low-rank structure in its unfolded form, meaning most of the information can be captured by a few dominant singular values.
  • Preserving Sharp Edges: While sparse approximation methods can sometimes smooth out images and lose fine details, matrix completion variants like CGIHT are better at preserving sharp edges and fine structures.

Key Results

  • Significant PSNR Improvement: The proposed non-convex matrix completion methods, initialized with SD, typically improve the Peak Signal-to-Noise Ratio (PSNR) by 2 dB to 5 dB compared to current state-of-the-art methods.
  • Visual Quality: Reconstructions using matrix completion are often visually indistinguishable from fully acquired data, with better preservation of sharp edges compared to sparse approximation.
  • Performance Across Datasets: These improvements were observed across diverse datasets, including urban, rural, and ground-based scenes.
  • Table 1 Summary: The paper provides a detailed table (Table 1) showing average PSNR values for various datasets and methods, highlighting the superiority of CGIHT and ASD initialized with SD in most cases.

Limitations and Future Work

  • Computational Cost: While non-convex matrix completion methods are shown to be effective, their computational cost is not explicitly detailed, and it's implied that they are more intensive than traditional methods.
  • Generalizability of Transforms: The paper notes that the choice of transform for sparse approximation (W2xDCT) is effective for natural scenes but doesn't explore other transforms extensively.
  • Specific Mosaic Arrangements: The paper mentions that some methods (like PPID) are limited to specific mosaic arrangements, implying that the proposed methods are more generally applicable.

Future work could involve exploring more efficient non-convex matrix completion algorithms, investigating optimal transform choices for sparse approximation across different scene types, and potentially developing methods that can adapt to various mosaic sensor designs.

Summary updated

Abstract

Snapshot mosaic multispectral imagery acquires an under-sampled data cube by acquiring a single spectral measurement per spatial pixel. Sensors which acquire p frequencies, therefore, suffer from severe 1/p undersampling of the full data cube. We show that the missing entries can be accurately imputed using non-convex techniques from sparse approximation and matrix completion initialised with traditional demosaicing algorithms. In particular, we observe the peak signal-to-noise ratio can typically be improved by 2 dB to 5 dB over current state-of-the-art methods when simulating a p = 16 mosaic sensor measuring both high and low altitude urban and rural scenes as well as ground-based scenes.
BibTeX
@inproceedings{antonucci2019multispectral,
  author = {Giancarlo A. Antonucci and Simon Vary and David Humphreys and Robert A. Lamb and Jonathan Piper and Jared Tanner},
  title = {{Multispectral Snapshot Demosaicing Via Non-Convex Matrix Completion}},
  booktitle = {DSW},
  year = {2019},
  url = {https://openreview.net/forum?id=t21iU1cIsK},
}
Figure