Super Resolving Unrolled Neural Networks for Remote Sensing

Johnathan Mulcahy-Stanislawczyk, Amber L. Dagel, Kin Long Kelvin Lee, Kevin Yarritu, Suyeon Shin, Eric A. Shields, Yufang Sun, Charles A. Bouman, Gregery T. Buzzard2024 · DOI

TL;DR

This paper introduces Consensus Multi-Image Super Resolution (CoMISR), a novel algorithm that leverages "unrolled" neural networks within a Consensus Equilibrium (CE) framework to achieve high-resolution remote sensing imagery from fewer input frames than traditional methods. This approach significantly reduces data acquisition needs and computational cost, making advanced super-resolution practical for real-time applications.

What Problem Does This Paper Address?

Remote sensing systems face a fundamental trade-off between system capabilities (like resolution) and practical constraints such as size, weight, and power (SWAP). While increasing optical aperture can improve resolution, it comes at a significant SWAP cost. Multi-Image Super Resolution (MISR) offers a computational alternative, combining multiple lower-resolution images into a single higher-resolution one. However, existing MISR techniques, like Drizzle, often require hundreds of images, drastically reducing temporal resolution and increasing data acquisition burdens, limiting their applicability. Traditional iterative optimization methods for super-resolution are computationally expensive and struggle with complex image models. Deep learning methods offer powerful image modeling but lack the theoretical robustness and explainability of iterative techniques. This paper aims to bridge this gap by developing a super-resolution method that combines the strengths of both iterative and deep learning approaches.

Methods

The core of this work is the application of Consensus Equilibrium (CE), a framework that generalizes iterative optimization techniques. CE allows the integration of "black-box" algorithms, such as neural network denoisers, as components within an iterative reconstruction process. The key innovation is "unrolling" the CE iterations, effectively creating a deep neural network where each layer corresponds to an iteration of the CE algorithm. This allows the neural network (specifically, the denoiser component) to be trained end-to-end, specializing it for the super-resolution task.

The authors adapted existing CE formulations and developed a Multi-Agent Consensus Equilibrium (MACE) approach for MISR. This involves treating each input image as a separate "agent" that contributes to a consensus solution. They also explored Generative Plug and Play (GPnP), a technique that uses generative models to sample from the posterior distribution of images, potentially offering better results than traditional least-squares minimization.

Datasets

The paper primarily uses simulated data for evaluation. This simulated data is generated by applying a Gaussian kernel to a ground truth image and adding Gaussian noise. For experimental validation, they used imagery of an Air Force 1951 resolution target. The low-resolution image size for this target was 128x128 pixels.

Key Insights and Ideas

  • Unrolling CE for Specialization: By unrolling the CE iterations into a neural network, the denoiser component can be specifically trained for the MISR task, leading to improved reconstruction quality.
  • CE as a Bridge: Consensus Equilibrium acts as a powerful framework to integrate the theoretical robustness of iterative methods with the expressive power of deep learning denoisers.
  • Order-of-Magnitude Reduction in Frames: The proposed CoMISR method demonstrates the ability to achieve high-quality reconstructions with significantly fewer input frames (e.g., 16 frames) compared to traditional methods like Drizzle, which might require hundreds.
  • Practical Speed: The reconstructions are produced "fast enough for practical usage," addressing a key limitation of many iterative super-resolution techniques.
  • Generative Approaches for Posterior Sampling: GPnP offers an alternative to direct optimization by sampling from the posterior distribution, potentially capturing more realistic image details.

Key Results

  • The CoMISR prototype produces high-quality MISR reconstructions using an order of magnitude fewer images than previous approaches.
  • Reconstructions are achieved fast enough for practical usage.
  • In simulated data experiments, CoMISR with DPIR regularization achieved competitive or superior results compared to SISR (Single-Image Super-Resolution) in terms of SSIM, reaching a peak SSIM of 0.7995 at an optimal regularization strength of 0.3.
  • For experimental data (Air Force 1951 resolution target), CoMISR with DPIR regularization showed reduced reconstruction artifacts and retained sharp edges compared to reconstructions without regularization.
  • Comparisons with Drizzle and ESPCN showed CoMISR outperforming both methods.

Limitations and Future Work

  • Training of Unrolled Network: While the paper demonstrated the feasibility of training an unrolled CE network and showed that the CE process is differentiable, they did not fully complete the training of such a network for the MISR application.
  • Computational Cost of Original CE: The paper notes that the original CE formulation can still involve solving large optimization problems in each iteration, which can be computationally intensive. The unrolling aims to mitigate this by enabling end-to-end training.
  • Future Work: The authors suggest that future research could focus on completing the end-to-end training of unrolled CE algorithms and exploring further applications of these techniques, particularly in conjunction with advances in generative AI. They also highlight the potential for this framework to be applied to a wider range of imaging systems and applications.

Summary updated

Abstract

In remote sensing systems, the capabilities of the system are constrained by the complex interactions between size, weight, and power (SWAP) of potential designs. In electro-optical (EO) systems, examples of these critical parameters include the system’s sensitivity and resolution. Those parameters can be increased by ever larger optical apertures and focal planes but at the cost of more SWAP. Multi-image super resolution (MISR) techniques allow resolution to be enhanced via computation rather than more sophisticated optical hardware. These algorithms combine multiple images together into a single, higher resolution image, trading temporal resolution and computation for spatial resolution. Fielded MISR techniques, such as Drizzle, can require several hundred images to create a single super resolved image, implying reduced temporal resolution, increased data acquisition load, and limiting mission applications. Iterative techniques, such as model-based image reconstruction and compressive sensing, have been shown to create super resolved images using fewer images than Drizzle. They do this by posing an optimization problem that balances accuracy between a highly accurate physical model and an image model. In the case of super resolution, the physical model is defined by the relation between low resolution input images and the desired high resolution output image. The image model encodes some assumptions about the super resolved image. These assumptions are meant to suppress reconstruction artifacts that arise due to deterministic physical model error, stochastic measurement noise, and potential undersampling. In practice, the performance of iterative methods are limited by imaging models compatible with optimization. Deep learning-based methods can effectively learn image models of arbitrary complexity, but lack the theoretical explainability and robustness of iterative techniques. Consensus equilibrium (CE) generalizes the iterative techniques beyond optimization, enabling blackbox algorithms such as traditional and neural image denoisers to be used as the image model. CE-based approaches retain much of the explainability and robustness of iterative techniques while allowing the expressiveness of machine learning image models to be used. Additionally, by unrolling iterations of CE with an embedded image denoiser, the image denoiser can be further trained and specialized to the specific application with potentially higher quality reconstructions. Under this project, we demonstrated the feasibility of training an unrolled neural network based upon CE. While we didn’t train one, we showed that the CE process is differentiable and its gradient can be tractably computed. We also explored the usage of a variants of CE akin to generative neural works. Most importantly, we applied the CE framework to a number of problems including non-blind deconvolution, upsampling, single-image super resolution, MISR, event-based sensing, and saturated deconvolution. Our MISR prototype creates high quality reconstructions with an order of magnitude fewer images than previous approaches and, critically, produces these reconstructions fast enough for practical usage.
BibTeX
@misc{mulcahystanislawczyk2024super,
  author = {Johnathan Mulcahy-Stanislawczyk and Amber L. Dagel and Kin Long Kelvin Lee and Kevin Yarritu and Suyeon Shin and Eric A. Shields and Yufang Sun and Charles A. Bouman and Gregery T. Buzzard},
  title = {{Super Resolving Unrolled Neural Networks for Remote Sensing}},
  year = {2024},
  doi = {10.2172/2480102},
}
Figure