Image Super-resolution Via Latent Diffusion: A Sampling-space Mixture Of Experts And Frequency-augmented Decoder Approach

Feng Luo, Jinxi Xiang, Jun Zhang, Xiao Han, Wei Yang

Abstract

The recent use of diffusion prior, enhanced by pre-trained text-image models, has markedly elevated the performance of image super-resolution (SR). To alleviate the huge computational cost required by pixel-based diffusion SR, latent-based methods utilize a feature encoder to transform the image and then implement the SR image generation in a compact latent space. Nevertheless, there are two major issues that limit the performance of latent-based diffusion. First, the compression of latent space usually causes reconstruction distortion. Second, huge computational cost constrains the parameter scale of the diffusion model. To counteract these issues, we first propose a frequency compensation module that enhances the frequency components from latent space to pixel space. The reconstruction distortion (especially for high-frequency information) can be significantly decreased. Then, we propose to use Sample-Space Mixture of Experts (SS-MoE) to achieve more powerful latent-based SR, which steadily improves the capacity of the model without a significant increase in inference costs. These carefully crafted designs contribute to performance improvements in largely explored 4x blind super-resolution benchmarks and extend to large magnification factors, i.e., 8x image SR benchmarks.

Introduction

While Latent Diffusion Models (LDMs) have revolutionized image generation, applying them to Super-Resolution (SR) presents two major hurdles. The first is the "Latent Bottleneck": to save on computation, LDMs operate in a compressed latent space. However, the autoencoders used for this compression often suffer from spectral bias—they capture low-frequency shapes well but discard high-frequency details like skin pores or fine textures, leading to reconstruction distortion.

The second hurdle is the "Scaling Wall": increasing the parameter count of a diffusion model typically leads to a linear or quadratic increase in training and inference costs. For high-resolution tasks, this quickly becomes prohibitively expensive for most hardware setups.

Methodology

To address these challenges, the researchers at Tencent AI Lab introduced a framework that combines frequency-aware refinement with a specialized Mixture of Experts (MoE) architecture.

Overview of the proposed latent diffusion framework featuring the SS-MoE UNet and frequency-augmented decoder.

1. Frequency-Augmented Decoder

Instead of relying on a standard VQGAN decoder, the authors implemented a Frequency Compensation Module. This module utilizes features from the original low-resolution (LR) image to ground the reconstruction. A refinement network using Adaptive Frequency Filtering (AFF) blocks is inserted after the decoder. This network operates in the frequency domain using Fourier Transforms to specifically inject high-frequency components back into the image, optimized by a focal frequency loss to match the ground truth spectrum.

2. Sample-Space Mixture of Experts (SS-MoE)

To scale model capacity efficiently, the framework employs a sparse MoE strategy:

  • Sampling MoE: Recognizing that noise levels vary drastically across the diffusion process (from $t=1000$ to $t=1$), the authors divide timesteps into stages, assigning a specific "expert" network to each. Only one expert is active at any given timestep, keeping inference costs constant.
  • Space MoE: Within the UNet, standard Feed-Forward Networks (FFNs) are replaced with spatial experts. A momentum update strategy is used during training to prevent "expert collapse," ensuring the model utilizes its full capacity.

Experimental Results

The model was trained on high-quality datasets including DIV2K and Flickr2K, and evaluated on both 4× blind SR and 8× non-blind SR benchmarks.

Perceptual Superiority and Efficiency

On the 4× DIV2K benchmark, the proposed method achieved an LPIPS score of 0.2821, significantly outperforming StableSR (0.3143) and the base LDM (0.3264). Lower LPIPS indicates images that are perceptually closer to the ground truth. Qualitatively, the model excels at restoring realistic textures where other models produce blurry patches.

Qualitative comparison of 4× blind super-resolution results across various state-of-the-art methods.

Furthermore, the model is remarkably efficient. Despite having ~605 million parameters, the total FLOPs (35.47T) are less than half of StableSR (86.27T), making it much more practical for real-world deployment.

Conclusion and Future Work

By specializing weights by noise level and moving refinement into the frequency domain, this approach achieves high-fidelity SR without the typical computational tax. While the model shows state-of-the-art performance, future work will focus on improving generalization to infinitely varied real-world degradations and potentially expanding latent feature channels to further mitigate the latent space bottleneck.

Summary updated

Abstract

The recent use of diffusion prior, enhanced by pre-trained text-image models, has markedly elevated the performance of image super-resolution (SR). To alleviate the huge computational cost required by pixel-based diffusion SR, latent-based methods utilize a feature encoder to transform the image and then implement the SR image generation in a compact latent space. Nevertheless, there are two major issues that limit the performance of latent-based diffusion. First, the compression of latent space usually causes reconstruction distortion. Second, huge computational cost constrains the parameter scale of the diffusion model. To counteract these issues, we first propose a frequency compensation module that enhances the frequency components from latent space to pixel space. The reconstruction distortion (especially for high-frequency information) can be significantly decreased. Then, we propose to use Sample-Space Mixture of Experts (SS-MoE) to achieve more powerful latent-based SR, which steadily improves the capacity of the model without a significant increase in inference costs. These carefully crafted designs contribute to performance improvements in largely explored 4x blind super-resolution benchmarks and extend to large magnification factors, i.e., 8x image SR benchmarks. The code is available at https://github.com/amandaluof/moe_sr.
BibTeX
@misc{luo2023image,
  author = {Feng Luo and Jinxi Xiang and Jun Zhang and Xiao Han and Wei Yang},
  title = {{Image Super-resolution Via Latent Diffusion: A Sampling-space Mixture Of Experts And Frequency-augmented Decoder Approach}},
  year = {2023},
  eprint = {2310.12004},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.12004},
}
Figure