Abstract
Image super-resolution (SR) is an ill-posed problem where regression-based methods often yield blurry results due to the "averaging" effect of standard loss functions like Mean Squared Error. While autoregressive (AR) models can generate high-fidelity textures by modeling pixel dependencies, their sequential nature makes them computationally prohibitive for high-resolution tasks. This paper introduces LAR-SR, a local autoregressive framework that achieves the expressive power of AR models with significantly reduced inference time. By decomposing the image into structural and textural components and employing a parallelized local sampling strategy, LAR-SR produces sharp, realistic textures at a fraction of the cost of global AR models.
Introduction: The Regression Dilemma
Most image super-resolution (SR) methods treat the task as a regression problem. By using losses like Mean Squared Error (MSE) or L1 to minimize the distance between predicted and ground-truth pixels, these models attempt to find a single "correct" answer. However, SR is inherently ill-posed: a single low-resolution (LR) image could correspond to many valid high-resolution (HR) versions. Regression models tend to "average" these possibilities, resulting in blurry, unrealistic outputs that lack fine textural detail.
Generative models like GANs and Normalizing Flows offer sharper details but face challenges such as training instability or restrictive architectural requirements. Autoregressive (AR) models provide a powerful alternative by modeling the probability of a pixel based on previous pixels, but they are traditionally too slow for practical SR applications because they sample pixel-by-pixel.
Methodology: The LAR-SR Framework
LAR-SR addresses the speed bottleneck of AR models through a two-stage pipeline that separates global structure from local texture.
Stage 1: Textural VQVAE (tex-VQVAE)
The model first separates the image into structural components and textural details. A standard regression model handles the coarse structure, providing a "structural anchor." Simultaneously, a Vector Quantized Variational AutoEncoder (VQVAE) learns a discrete "codebook" specifically for fine-grained textures. This ensures the subsequent AR model only needs to predict fine details rather than the entire image structure.
Stage 2: Local Autoregressive (LAR) Module
The core innovation is the Local Autoregressive module. Instead of a global AR model that predicts the entire image sequentially, LAR-SR divides the image into non-overlapping patches (e.g., 4x4).
- Parallelism: The model samples the "first" pixel of every patch simultaneously across the entire image. It then moves to the "second" pixel of every patch, and so on.
- Local Dependency: Within each patch, pixels are predicted sequentially. This allows the model to capture complex relationships between adjacent pixels—essential for realistic texture—while benefiting from massive parallelism.
This "same-index parallel sampling" trick reduces time complexity from being dependent on the total number of pixels ($H \times W$) to being dependent only on the number of pixels in a single small patch ($s \times s$).
Key Results and Performance
LAR-SR was evaluated on standard benchmarks including DIV2K for general SR and CelebA/FFHQ for face SR.
- Perceptual Quality: LAR-SR outperformed GAN and Flow-based baselines on the LPIPS metric, which measures human-like visual perception. On DIV2K, it achieved an LPIPS of 0.114, surpassing ESRGAN (0.124) and SRFlow (0.121).
- Efficiency Gains: Compared to a traditional global AR baseline (VQVAE + PixelCNN), LAR-SR reduced sampling time from 230.7 seconds to just 1.2 seconds for the same resolution—a reduction to approximately 0.52% of the original time.
- Model Variants: An attention-based variant, LAR-attn-SR, further reduced the model size from 62.1M parameters to 10.1M while maintaining or slightly improving performance.
Conclusion
LAR-SR demonstrates that the high representative power of autoregressive models can be harnessed for super-resolution without the traditional computational burden. By focusing on local texture and parallelizing the sequential sampling process, the framework bridges the gap between the blurry outputs of regression models and the slow inference of global generative models. While still slower than pure regression models, the 100x speedup over traditional AR methods marks a significant step toward practical, high-fidelity generative super-resolution.
Summary updated