Image super-resolution with PCA Reduced generalized Gaussian mixture models

Dang-Phuong-Lan Nguyen, Johannes Hertrich, Jean–François Aujol, Yannick BerthoumieuHAL 2024

TL;DR

This paper introduces a novel approach to single-image super-resolution (SISR) by combining generalized Gaussian mixture models (GGMMs) with Principal Component Analysis (PCA) for dimensionality reduction. This method, called PCA-GGMM, effectively handles high-dimensional data and is invariant to affine contrast changes, outperforming existing methods on synthetic and material microstructure images.

What Problem Does This Paper Address?

Single-image super-resolution (SISR) aims to reconstruct a high-resolution (HR) image from a single low-resolution (LR) input image. A significant challenge in this field is that the exact super-resolution operator (how the LR image was generated from the HR image) is often unknown, and the contrast between LR and HR images can vary. Traditional methods, like those using Gaussian Mixture Models (GMMs), struggle with the high dimensionality of patch-based data, leading to computational instability and intractability. While generalized Gaussian mixture models (GGMMs) offer more flexibility in modeling data distributions, they also face these high-dimensionality issues. This paper addresses these limitations by proposing a method that is robust to unknown operators and affine contrast changes, while also being computationally efficient for high-dimensional data.

Methods

The core of this work lies in two main innovations:

  • MMSE-GGMM: The authors extend the work of Sandeep and Jacob [10] by using a joint GGMM for modeling high- and low-resolution patches. They employ a Minimum Mean Squared Error (MMSE) estimator for reconstructing HR patches, which is known to be invariant to affine contrast changes. This approach avoids the need to explicitly learn the super-resolution operator.
  • PCA-GGMM: To combat the computational challenges arising from high-dimensional data (especially with 3D images), they integrate Principal Component Analysis (PCA) into the GGMM framework. This dimensionality reduction technique, applied to the low-dimensional latent space, results in a PCA-reduced generalized Gaussian distribution. An Expectation-Maximization (EM) algorithm is derived to estimate the parameters of this PCA-GGMM model, utilizing a "Half-Quadratic splitting" method to handle the complex optimization.

Datasets

The paper evaluates their methods on:

  • Synthetic datasets: Standard images like Gold Hill, Camera-Man, and Barbara.
  • Material microstructure datasets: Two types of 3D images: Fontainebleau sandstone ("FS") and a composite ("SiC Diamonds"). These datasets are described as having diverse levels of complexity.

The authors generated low-resolution images from high-resolution originals by applying a blur operator (Gaussian kernel) and a downsampling operator, with added white Gaussian noise.

Key Insights and Ideas

  • GGMMs for richer patch modeling: GGMMs are more flexible than GMMs in capturing the diverse shapes of image patch distributions, leading to better reconstruction quality.
  • MMSE for contrast invariance: The MMSE estimator inherently handles affine contrast and brightness changes, a crucial advantage when dealing with real-world image acquisition.
  • PCA as a dimensionality reducer for GGMMs: Applying PCA to the latent space of GGMMs significantly reduces computational complexity and improves numerical stability, making the models applicable to high-dimensional data (like 3D microstructures).
  • PCA-GGMM as a powerful combination: The synergy between PCA for dimensionality reduction and GGMM for robust modeling leads to a highly effective super-resolution framework.

Key Results

  • Improved PSNR: The proposed MMSE-GGMM method consistently achieved higher Peak Signal-to-Noise Ratio (PSNR) values compared to MMSE-GMM and EPLL methods, especially on material data where contrast changes are more pronounced. For instance, on material data, their method showed PSNRs 1 dB higher than EPLL with a learned operator.
  • PCA-GGMM performance: The PCA-GGMM model demonstrated results comparable to, and sometimes slightly better than, the MMSE-GGMM, while offering significant computational advantages for high-dimensional data. For 3D material data, PCA-GGMM achieved competitive PSNRs.
  • Operator-free reconstruction: The MMSE-based approaches (MMSE-GGMM and PCA-GGMM) do not require prior knowledge of the super-resolution operator, unlike EPLL methods.

Limitations and Future Work

  • Computational complexity of M-step: While PCA-GGMM reduces the complexity of the E-step, the M-step can still be computationally intensive due to the GGMM's complexity, although it operates in a lower-dimensional space.
  • Deep learning approaches: The authors acknowledge the effectiveness of deep learning methods for super-resolution and suggest that future work could explore integrating deep learning approaches with their proposed models, especially for high magnification factors and high-dimensional data.
  • Extension to elliptical distributions: The PCA-reduced generalized Gaussian distribution could be generalized to other elliptical distributions, such as the Student-t distribution.

Summary updated

Abstract

Single Image Super-Resolution algorithms based on patches have been noticed and widely used over the past decade. Recently, generalized Gaussian mixture models (GGMMs) have been shown to be a suitable tool for many image processing problems because of the flexible shape parameter. In this work, we first propose to use a joint GGMM learned from concatenated vectors of high-and low-resolution training patches. For each low-resolution patch, we compute the minimum mean square error (MMSE) estimator and generate the high-resolution image by averaging these estimates. We select the MMSE approach using GGMM as the method is invariant to affine contrast change and also invariant to a linear super-resolution operator. Unfortunately, the large dimension of the concatenated high-and low-resolution patches leads to instabilities and an intractable computational effort when estimating the parameters of the GGMM. Thus, we propose to combine a GGMM with a principal component analysis and derive an EM algorithm for estimating the parameters of the arising model. We demonstrate the performance of our model by numerical examples on synthetic and real images of materials' microstructures.
BibTeX
@inproceedings{nguyen2024image,
  author = {Dang-Phuong-Lan Nguyen and Johannes Hertrich and Jean–François Aujol and Yannick Berthoumieu},
  title = {{Image super-resolution with PCA Reduced generalized Gaussian mixture models}},
  booktitle = {HAL},
  year = {2024},
}
Figure