Rethinking Image Super Resolution from Long-Tailed Distribution Learning Perspective

Yuanbiao Gou, Peng Hu, Jiancheng Lv, Hongyuan Zhu, Xi Peng

Introduction: The Challenge of High-Frequency Details

In the field of Single Image Super-Resolution (SR), a persistent challenge remains: while models are highly effective at reconstructing smooth, low-frequency (LF) regions like clear skies or flat walls, they often struggle to sharpen complex high-frequency (HF) textures such as hair, fabric, or fine edges. Traditional approaches to this problem involve increasing model depth or adding specialized edge-enhancement modules. However, these methods often fail to address the underlying cause of the performance gap.

This paper identifies the root cause as a long-tailed distribution problem. In natural images, smooth areas vastly outnumber detailed ones. Consequently, standard training objectives are dominated by the "easy" pixels, leading models to overfit simple regions while underperforming on the critical details that define visual quality.

The Long-Tailed Distribution Problem

The authors argue that SR suffers from a "twin fitting problem" inherent in the pixel distribution of natural images. When analyzing the absolute difference between a low-resolution (LR) input and its high-resolution (HR) ground truth, the majority of pixels exhibit very small differences (the "head" of the distribution). Only a tiny fraction of pixels—representing edges and textures—constitute the "long tail" of large differences.

Visualization of the long-tailed distribution of pixel-wise errors in super-resolution.

Because standard loss functions treat every pixel equally, the model's learning process is biased toward the easy majority. This results in a model that is an expert at reconstructing "boring" regions but remains a novice at the intricate details that human eyes prioritize.

Focal Pixel Learning (FPL)

To address this imbalance, the authors propose Focal Pixel Learning (FPL), a model-agnostic training strategy that rebalances pixel importance. FPL bridges the gap between low-level SR and high-level long-tailed learning techniques (like Focal Loss used in object detection) through three main components:

  1. Under-sampling via Residuals: The model is trained to predict the residual difference between the ground truth and a bicubic interpolation, effectively muting easy-to-predict smooth areas.
  2. Dual Structure Priors:
    • Static Prior ($y_{sp}$): Identifies high-frequency areas based on the difference between the original image and its bicubic upscaling.
  3. Learnable Prior ($y_{lp}$): Dynamically weights pixels based on the model's current fitting error during training.
  4. Weighting Function: An exponential function uses these priors to adjust the loss for each pixel, ensuring gradients are balanced between the easy majority and the difficult minority.
The proposed Focal Pixel Learning (FPL) training framework.

Crucially, FPL is a training-time intervention. The resulting models can be deployed without any additional computational overhead during inference.

Experimental Results

The FPL strategy was evaluated across multiple CNN architectures (FSRCNN, CARN, SRResNet, MSRN) and the Transformer-based SwinIR on six benchmark datasets, including Set5, Manga109, and Urban100.

  • Universal Improvement: FPL consistently improved PSNR and SSIM across all tested models and datasets.
  • Boosting Lightweight Models: The gains were particularly significant for smaller architectures. For instance, the tiny FSRCNN saw a boost of up to +0.63dB on the Manga109 dataset, allowing it to compete with much larger models.
  • Efficiency: An SRResNet trained with FPL (1.5M parameters) outperformed the significantly larger MSRN (6.1M parameters) on several benchmarks.
  • Visual Fidelity: Qualitative results demonstrate that FPL-trained models produce significantly sharper edges and more realistic textures compared to those using standard training methods.

Conclusion

By reframing Super-Resolution as a long-tailed distribution problem, Focal Pixel Learning provides a powerful, model-agnostic method for improving image quality. It forces models to master difficult high-frequency details without requiring larger architectures or increasing inference costs. Future work may extend this perspective to perceptual and adversarial losses to further enhance visual realism.

Summary updated

Abstract

Existing studies have empirically observed that the resolution of the low-frequency region is easier to enhance than that of the high-frequency one. Although plentiful works have been devoted to alleviating this problem, little understanding is given to explain it. In this paper, we try to give a feasible answer from a machine learning perspective, i.e., the twin fitting problem caused by the long-tailed pixel distribution in natural images. With this explanation, we reformulate image super resolution (SR) as a long-tailed distribution learning problem and solve it by bridging the gaps of the problem between in low- and high-level vision tasks. As a result, we design a long-tailed distribution learning solution, that rebalances the gradients from the pixels in the low- and high-frequency region, by introducing a static and a learnable structure prior. The learned SR model achieves better balance on the fitting of the low- and high-frequency region so that the overall performance is improved. In the experiments, we evaluate the solution on four CNN- and one Transformer-based SR models w.r.t. six datasets and three tasks, and experimental results demonstrate its superiority.
BibTeX
@inproceedings{gou2023rethinking,
  author = {Yuanbiao Gou and Peng Hu and Jiancheng Lv and Hongyuan Zhu and Xi Peng},
  title = {{Rethinking Image Super Resolution from Long-Tailed Distribution Learning Perspective}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.01377},
}
Figure