Learning Coupled Dictionaries from Unpaired Data for Image Super-Resolution

Longguang Wang, Juncheng Li, Yingqian Wang, Qingyong Hu, Yulan GuoCVPR 2024 · DOI

Abstract

Unsupervised image super-resolution (SR) typically relies on synthesizing "pseudo" low-resolution (LR) images to create paired training data. However, this process often introduces artifacts and fails to capture real-world degradation complexity. This paper introduces DictSR, a novel framework that learns coupled dictionaries from unpaired data. By mapping high-resolution (HR) and LR images into a shared, discrete latent space, DictSR avoids the pitfalls of image-space synthesis. The method achieves state-of-the-art perceptual quality with a computationally efficient 18-layer architecture.

The Challenge of Unpaired Super-Resolution

Deep learning models for Super-Resolution (SR) generally require paired high-resolution (HR) and low-resolution (LR) images for training. In practical scenarios, obtaining such pairs is often impossible. Existing unsupervised methods attempt to bridge this gap by using Generative Adversarial Networks (GANs) or diffusion models to synthesize "pseudo" LR images.

Unfortunately, the image space is massive and complex. Synthesized images frequently suffer from mode collapse (lack of variety) or low content fidelity, limiting the model's effectiveness on real-world data. DictSR addresses this by shifting the problem from image synthesis to representation matching in a compact proxy space.

Comparison of previous image-synthesis approaches versus the proposed latent-code matching approach.

Methodology: Coupled Dictionaries

The core of the DictSR architecture is a set of coupled dictionaries that represent images as combinations of discrete "atoms."

  1. Shared Latent Space: The model learns an HR dictionary ($D^{HR}$) and an LR dictionary ($D^{LR}$). During training, it ensures that an HR patch and its corresponding LR version share the same "spatial code" (index) in their respective dictionaries. This forces the model to ignore pixel-level noise and focus on underlying structure.
  2. Hierarchical LR Representation: Real-world LR images are a mix of scene structure and external "damage" (blur, noise, compression). To handle this, the authors split the LR dictionary into a content dictionary ($D_{con}^{LR}$) and a degradation dictionary ($D_{deg}$). This decoupling allows the model to strip away degradation more effectively.
  3. Efficient Autoencoder Framework: The system uses encoders to map images to dictionary indices and decoders to reconstruct them. During the test phase, an LR image is encoded using the LR dictionaries, but the resulting indices are used to pull "clean" features from the HR dictionary to reconstruct the final SR result.
  4. Computational Efficiency: Because the dictionaries handle the heavy lifting of feature representation, the neural network itself is remarkably shallow—just 18 layers—making it much faster and lighter than competing models.

Key Results and Performance

DictSR was evaluated on major benchmarks for real-world unpaired SR, including AIM-RWSR and NTIRE-RWSR, using DIV2K as the source for high-resolution ground truth.

  • Superior Perceptual Quality: On the AIM-RWSR dataset, DictSR achieved an LPIPS score of 0.259, significantly outperforming other unpaired methods like CycleGAN (0.476) or DASR (0.336). A lower LPIPS score indicates images that look more natural to human observers.
  • Competitive Accuracy: It maintained high pixel-wise accuracy (22.46 dB PSNR), outperforming many GAN-based methods that often sacrifice fidelity for sharpness.
  • Efficiency: DictSR requires only 129G MACs, which is less than half the computational cost of popular models like Real-ESRGAN (291G).
Qualitative comparison of DictSR against other state-of-the-art super-resolution methods on real-world images.

Conclusion and Future Work

By leveraging discrete latent spaces and coupled dictionaries, DictSR provides a robust and efficient solution for unsupervised super-resolution. It avoids the complexities of high-dimensional image synthesis while maintaining superior perceptual quality. While currently focused on x4 super-resolution, the dictionary-coupling approach holds promise for other image restoration tasks, such as denoising or deblurring, where unpaired data remains a major bottleneck.

Summary updated

BibTeX
@inproceedings{wang2024learning,
  author = {Longguang Wang and Juncheng Li and Yingqian Wang and Qingyong Hu and Yulan Guo},
  title = {{Learning Coupled Dictionaries from Unpaired Data for Image Super-Resolution}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.02429},
}
Figure