Multi-task Gaussian Process Regression-based Image Super Resolution

Xinwei Jiang, Jie Yang, Лей Ма, Yiping Yang2015 · DOI

TL;DR

This paper introduces MTGPSR, a novel image super-resolution (SR) framework that leverages Multi-Task Gaussian Process (MTGP) regression. By treating each pixel prediction as a separate task and explicitly modeling the correlations between these tasks, MTGPSR aims to reconstruct more accurate and natural-looking high-resolution images, outperforming existing methods in structural similarity.

What Problem Does This Paper Address?

Image Super-Resolution (SR) is the task of reconstructing a high-resolution (HR) image from a low-resolution (LR) counterpart. This is a challenging ill-posed problem because a single LR image often contains insufficient information to perfectly recover the lost high-frequency details. Existing SR methods fall into categories like interpolation-based (fast but blurry), reconstruction-based (rely on image priors but struggle with large scaling factors), and example learning-based (achieve state-of-the-art but can be computationally intensive). While Gaussian Process Regression (GPR) has shown promise in SR due to its non-parametric nature, previous GPR-based methods often treated each pixel prediction independently, ignoring the inherent correlations within an image patch. This paper addresses this limitation by proposing a framework that explicitly models these inter-pixel correlations.

Methods

The core innovation is the application of Multi-Task Gaussian Process (MTGP) regression to SR. Instead of using independent GPR models for each pixel in an HR patch, MTGPSR treats each pixel prediction as a distinct "task." MTGP allows for the explicit modeling of correlations between these tasks (pixels) through an inter-task covariance matrix ($K_f$). This allows information from one pixel's prediction to inform others, leading to more coherent and natural results.

The MTGPSR framework involves:

  1. Training Stage:
    • Constructing LR-HR patch pairs by downsampling HR images.
  2. Clustering the LR patches using K-means.
  3. Learning a separate MTGP model for each cluster to capture the specific LR-HR mapping within that cluster.
  4. Predicting Stage:
    • Given an LR image, overlapping sample it to obtain test patches.
  5. Classify each test LR patch into one of the learned clusters using K-NN.
  6. Use the corresponding MTGP model to predict the HR patch subspace.
  7. Reconstruct the final HR image by aggregating the predicted HR patches using an average weighting scheme.

The paper also details the optimization of MTGP hyper-parameters using an Expectation-Maximization (EM) algorithm and adopts a squared exponential covariance function for the GPs.

Datasets

The paper evaluates its method on the classic Set5 and Set14 datasets, which are standard benchmarks for image super-resolution research. The training data was derived from HR images provided by reference [15].

Key Insights and Ideas

  • Pixel Correlation is Key: The most significant insight is that treating pixel predictions as correlated tasks within an MTGP framework significantly improves the naturalness and accuracy of super-resolved images, moving beyond independent pixel estimations.
  • MTGP for Structure: By modeling inter-task correlations, MTGP can implicitly learn and adapt to local image structures, leading to better performance on textured or complex regions.
  • Clustering for Efficiency: Clustering LR patches allows the framework to learn specialized MTGP models for different types of image patches, reducing the computational burden compared to learning a unique model for every single patch.
  • SSIM as a Better Metric: While PSNR is a common metric, the paper highlights that MTGPSR excels in Structural Similarity (SSIM), suggesting its outputs are more perceptually pleasing to human observers.

Key Results

  • MTGPSR achieves competitive PSNR values, often outperforming baseline methods like Bi-cubic, ScSR, GPR, and PGPR.
  • Crucially, MTGPSR consistently prevails over all compared methods in terms of SSIM, indicating superior perceptual quality.
  • The authors note that while their PSNR might be slightly lower than SLGPR on some images, their SSIM performance is superior.
  • The paper uses a smaller patch size (3x3) for MTGPSR compared to some other methods (e.g., SLGPR uses 7x7), which helps manage training time, yet still achieves competitive or better results.

Limitations and Future Work

  • Training Time: The paper acknowledges that increasing the patch size for MTGPSR can make the training time prohibitively long (over a week on CPU for larger sizes).
  • Future Work: The authors plan to focus on reducing the training time of MTGP and exploring its application to other image processing tasks like denoising and enhancement.

Summary updated

Abstract

Image super resolution (SR) aims at recovering the missing high frequency details from single image or multiple images. Existing SR methods can be divided into three categories: interpolation-based, reconstructionbased and example learning-based. Our paper focuses on the third category. Example learning-based SRmethods [6] utilize the LR-HR image pair to infer the missing high-frequency details in the LR image and achieve state-of-the-art performance.Recently, in the field of example learningbased SR, more and more researchers resort to learn the LR-HR relationship directly, i.e. y = f (x), where x is the input LR image feature, y is the targeted HR image and f is the mapping function that transforms the LR feature into HR image. Instead of commonly used parametric models, non-parametric methods [3], especially gaussian process regression (GPR)-related methods [2, 4, 5] begin to emerge in the SR field. However, previous GPR-based SR methods simply learn all the GPR models independently and ignore the correlation between them. On the other hand, each pixel prediction can be treated as a task, so that inferring a HR patch can be regarded as a multi-task problem. In this paper, we focus on the multi-task gaussian process (MTGP) regression and apply it to superresolution problem. We first give a brief overview of MTGP proposed in [1]. Then we study how SR problem corresponds to MTGP and propose the multi-task gaussian process super-resolution (MTGPSR) framework. MTGP tries to solve the following problem: Given N distinct inputs x1, ...,xN we define the complete set of responses for M tasks as y = (y11, ...,yN1, ...,y12, ...,yN2, ...,y1M , ...,yNM) , where yi j is the response for the jth task on the ith input xi. We also denote the N ×M matrix Y such that y = vecY . Given a set of observations yo, which is a subset of y, we wish to predict the unobserved values of yu of some input points for some tasks. MTGP wishes to learn M related latent functions { fl} by placing a GP prior over { fl} and directly induce correlations between tasks. Assuming that the GPs have zero mean we define ⟨ fl(x) fk(x ′) ⟩ = K f lkk x(x,x′) (1)
BibTeX
@misc{jiang2015multi,
  author = {Xinwei Jiang and Jie Yang and Лей Ма and Yiping Yang},
  title = {{Multi-task Gaussian Process Regression-based Image Super Resolution}},
  year = {2015},
  doi = {10.5244/c.29.151},
}
Figure