VarSR: Variational Super-Resolution Network for Very Low Resolution Images

Sangeek Hyun, Jae-Pil HeoECCV 2020 · OpenReview

Abstract

Single image super-resolution (SR) is an ill-posed problem where multiple high-resolution (HR) images can be matched to one low-resolution (LR) image. This many-to-one nature is particularly magnified when super-resolving from very low-dimensional domains, such as 8x8 resolution, where detailed HR information is nearly impossible to recover deterministically. Most existing methods are optimized for deterministic generation, resulting in one-to-one correspondences that often produce blurry "average" images. We propose VarSR, a Variational Super-Resolution Network that matches the latent distributions of LR and HR images. By drawing samples from a learned common latent distribution, VarSR generates diverse and perceptually plausible SR images. Experimental results demonstrate that VarSR produces more accurate and visually realistic outputs compared to deterministic techniques.

Introduction

Single Image Super-Resolution (SISR) is fundamentally ill-posed: a single low-resolution image could have originated from numerous distinct high-resolution sources. Standard deep learning models typically employ a deterministic approach, mapping one LR input to exactly one SR output. To minimize reconstruction error (like MSE), these models often output the mean of all possible HR candidates, which manifests as blurriness and a lack of fine detail.

This limitation is most severe at very low resolutions (VLR), such as 8x8 pixels, where high-frequency information is almost entirely lost. Instead of a single "best guess," a robust SR system should be capable of generating a variety of realistic possibilities that are all consistent with the low-resolution input.

Comparison of deterministic models vs. VarSR-Net in handling the many-to-one mapping problem.

Methodology: VarSR-Net

The Variational Super-Resolution Network (VarSR-Net) adopts a stochastic framework inspired by Conditional Variational Autoencoders (CVAE) to model the inherent uncertainty in VLR SR.

Architecture and Training

The system utilizes a dual-encoder strategy during training. An HR encoder ($E_H$) extracts features from the ground-truth high-resolution image, while an LR encoder ($E_L$) attempts to predict these features from the low-resolution input alone. The model is trained to minimize the KL-divergence between the distributions produced by $E_H$ and $E_L$, effectively forcing the LR encoder to learn a latent space that captures the missing HR details.

The final objective function is a hybrid loss comprising:

  • Pixel Loss ($L_2$): Ensures spatial and structural accuracy.
  • KL Divergence Loss: Aligns the LR and HR latent distributions.
  • Adversarial Loss (WGAN-GP): Encourages the generator to produce sharp, realistic textures rather than blurry averages.

Inference

During inference, the HR image is unavailable. The model draws multiple latent samples ($z$) from the distribution predicted by the LR encoder. Each sample is concatenated with the LR features and passed through the Generator ($G$), producing a set of diverse, valid high-resolution candidates.

The training and inference pipelines for VarSR-Net, highlighting the distribution matching and stochastic sampling process.

Experimental Evaluation

The authors evaluated VarSR across three datasets to test perceptual quality, identity consistency, and semantic reliability.

  1. CelebA (Faces): For 8x upscaling (8x8 to 64x64). VarSR achieved superior LPIPS (perceptual similarity) and FaceNet (identity verification) scores compared to deterministic baselines like SRGAN.
  2. MNIST (Digits): Downsampled to 6x6. VarSR achieved a classification accuracy of 92.79% (using the "best of 5" samples), significantly higher than the 84.74% achieved by deterministic models.
  3. License Plate (LP) Characters: A dataset of character-level images downsampled to 2x4. VarSR successfully recovered legible characters from extremely degraded inputs.

Discussion and Conclusion

VarSR effectively addresses the "many-to-one" nature of super-resolution by embracing stochasticity. Beyond generating diverse images, the learned latent space allows for attribute editing (e.g., modifying facial expressions) while maintaining consistency with the LR input.

While the sampling process requires multiple forward passes—making it more computationally intensive than deterministic models—the gain in perceptual realism and semantic accuracy is substantial for very low-resolution tasks. Future work may explore scaling this approach to higher resolutions and standardizing evaluation metrics for diverse generative SR.

Summary updated

Abstract

As is well known, single image super-resolution (SR) is an ill-posed problem where multiple high resolution (HR) images can be matched to one low resolution (LR) image due to the difference in their representation capabilities. Such many-to-one nature is particularly magnified when super-resolving with large upscaling factors from very low dimensional domains such as 8 $$\times $$ 8 resolution where detailed information of HR is hardly discovered. Most existing methods are optimized for deterministic generation of SR images under pre-defined objectives such as pixel-level reconstruction and thus limited to the one-to-one correspondence between LR and SR images against the nature. In this paper, we propose VarSR, Variational Super Resolution Network, that matches latent distributions of LR and HR images to recover the missing details. Specifically, we draw samples from the learned common latent distribution of LR and HR to generate diverse SR images as the many-to-one relationship. Experimental results validate that our method can produce more accurate and perceptually plausible SR images from very low resolutions compared to the deterministic techniques.
BibTeX
@inproceedings{hyun2020varsr,
  author = {Sangeek Hyun and Jae-Pil Heo},
  title = {{VarSR: Variational Super-Resolution Network for Very Low Resolution Images}},
  booktitle = {European Conference on Computer Vision},
  year = {2020},
  url = {https://openreview.net/forum?id=P1PRRniFr_l},
}
Figure