LAR-SR: A Local Autoregressive Model for Image Super-Resolution

Baisong Guo, Xiaoyun Zhang, Haoning Wu, Yu Wang, Ya Zhang, Yanfeng Wang

Abstract

Image super-resolution (SR) is an ill-posed problem where regression-based methods often yield blurry results due to the "averaging" effect of standard loss functions like Mean Squared Error. While autoregressive (AR) models can generate high-fidelity textures by modeling pixel dependencies, their sequential nature makes them computationally prohibitive for high-resolution tasks. This paper introduces LAR-SR, a local autoregressive framework that achieves the expressive power of AR models with significantly reduced inference time. By decomposing the image into structural and textural components and employing a parallelized local sampling strategy, LAR-SR produces sharp, realistic textures at a fraction of the cost of global AR models.

Introduction: The Regression Dilemma

Most image super-resolution (SR) methods treat the task as a regression problem. By using losses like Mean Squared Error (MSE) or L1 to minimize the distance between predicted and ground-truth pixels, these models attempt to find a single "correct" answer. However, SR is inherently ill-posed: a single low-resolution (LR) image could correspond to many valid high-resolution (HR) versions. Regression models tend to "average" these possibilities, resulting in blurry, unrealistic outputs that lack fine textural detail.

Comparison of regression-based (averaging) and autoregressive (sequential) modeling for super-resolution.

Generative models like GANs and Normalizing Flows offer sharper details but face challenges such as training instability or restrictive architectural requirements. Autoregressive (AR) models provide a powerful alternative by modeling the probability of a pixel based on previous pixels, but they are traditionally too slow for practical SR applications because they sample pixel-by-pixel.

Methodology: The LAR-SR Framework

LAR-SR addresses the speed bottleneck of AR models through a two-stage pipeline that separates global structure from local texture.

Stage 1: Textural VQVAE (tex-VQVAE)

The model first separates the image into structural components and textural details. A standard regression model handles the coarse structure, providing a "structural anchor." Simultaneously, a Vector Quantized Variational AutoEncoder (VQVAE) learns a discrete "codebook" specifically for fine-grained textures. This ensures the subsequent AR model only needs to predict fine details rather than the entire image structure.

Stage 2: Local Autoregressive (LAR) Module

The core innovation is the Local Autoregressive module. Instead of a global AR model that predicts the entire image sequentially, LAR-SR divides the image into non-overlapping patches (e.g., 4x4).

The two-stage LAR-SR architecture featuring textural VQVAE and the Local Autoregressive (LAR) module.
  • Parallelism: The model samples the "first" pixel of every patch simultaneously across the entire image. It then moves to the "second" pixel of every patch, and so on.
  • Local Dependency: Within each patch, pixels are predicted sequentially. This allows the model to capture complex relationships between adjacent pixels—essential for realistic texture—while benefiting from massive parallelism.

This "same-index parallel sampling" trick reduces time complexity from being dependent on the total number of pixels ($H \times W$) to being dependent only on the number of pixels in a single small patch ($s \times s$).

Key Results and Performance

LAR-SR was evaluated on standard benchmarks including DIV2K for general SR and CelebA/FFHQ for face SR.

  • Perceptual Quality: LAR-SR outperformed GAN and Flow-based baselines on the LPIPS metric, which measures human-like visual perception. On DIV2K, it achieved an LPIPS of 0.114, surpassing ESRGAN (0.124) and SRFlow (0.121).
  • Efficiency Gains: Compared to a traditional global AR baseline (VQVAE + PixelCNN), LAR-SR reduced sampling time from 230.7 seconds to just 1.2 seconds for the same resolution—a reduction to approximately 0.52% of the original time.
  • Model Variants: An attention-based variant, LAR-attn-SR, further reduced the model size from 62.1M parameters to 10.1M while maintaining or slightly improving performance.

Conclusion

LAR-SR demonstrates that the high representative power of autoregressive models can be harnessed for super-resolution without the traditional computational burden. By focusing on local texture and parallelizing the sequential sampling process, the framework bridges the gap between the blurry outputs of regression models and the slow inference of global generative models. While still slower than pure regression models, the 100x speedup over traditional AR methods marks a significant step toward practical, high-fidelity generative super-resolution.

Summary updated

Abstract

Previous super-resolution (SR) approaches often formulate SR as a regression problem and pixel wise restoration, which leads to a blurry and unreal SR output. Recent works combine adversarial loss with pixel-wise loss to train a GAN-based model or introduce normalizing flows into SR problems to generate more realistic images. As another powerful generative approach, autoregressive (AR) model has not been noticed in low level tasks due to its limitation. Based on the fact that given the structural in-formation, the textural details in the natural images are locally related without long term dependency, in this paper we propose a novel autoregressive model-based SR approach, namely LAR-SR, which can efficiently generate realistic SR images using a novel local autoregressive (LAR) module. The proposed LAR module can sample all the patches of textural components in parallel, which greatly reduces the time consumption. In addition to high time efficiency, it is also able to leverage contextual information of pixels and can be optimized with a consistent loss. Experimental results on the widely-used datasets show that the proposed LAR-SR approach achieves superior performance on the vi-sual quality and quantitative metrics compared with other generative models such as GAN, Flow, and is competitive with the mixture generative model.
BibTeX
@inproceedings{guo2022lar,
  author = {Baisong Guo and Xiaoyun Zhang and Haoning Wu and Yu Wang and Ya Zhang and Yanfeng Wang},
  title = {{LAR-SR: A Local Autoregressive Model for Image Super-Resolution}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
  doi = {10.1109/cvpr52688.2022.00195},
}
Figure