Pixel-Level Self-Paced Learning For Super-Resolution

Wei Lin, Junyu Gao, Qi Wang, Xuelong Li

Executive Summary

Training state-of-the-art deep networks for Single Image Super-Resolution (SISR) is notoriously slow, often taking days to converge. This paper introduces Pixel-Level Self-Paced Learning (PSPL), a training strategy that accelerates convergence and improves final image quality by dynamically weighting the importance of individual pixels based on their reconstruction difficulty.

Introduction: The Training Bottleneck in Super-Resolution

As Super-Resolution (SR) models grow deeper and more complex (e.g., EDSR, DBPN), they become increasingly difficult to train. These models often require massive computational resources and long training cycles—sometimes up to a week for a single model—to avoid getting stuck in local optima.

While "Self-Paced Learning" (SPL) is a known technique in machine learning to improve training by presenting "easy" samples before "hard" ones, it is typically applied at the image level. In the context of SR, where the goal is to recover fine-grained spatial details, an entire image might contain both easy-to-reconstruct smooth regions and hard-to-reconstruct textures. The authors argue that traditional image-level SPL is too coarse for SR; instead, the "pacing" needs to happen at the pixel level.

The PSPL Framework

The authors propose a four-step training pipeline called PSPL that modifies how the loss function "sees" the image during training:

  1. SR Generation: The model predicts a high-resolution (SR) image from a low-resolution input.
  2. Similarity Map Production: The system compares the predicted SR image to the Ground Truth (HR) image using the Structural Similarity Index (SSIM). This creates a map where each value represents how well a local patch was reconstructed.
  3. Attention Map Calculation: Using a Gaussian function, the similarity map is converted into an Attention Map ($M_a$). Early in training, the model assigns much higher weights to pixels with low similarity (the "hard" pixels).
  4. Modified Loss Function: The loss is calculated by performing element-wise multiplication between the Attention Map and the images. This forces the optimizer to prioritize correcting the most "difficult" pixels first.
The PSPL training pipeline: from SR generation to the modified loss function using an attention map.

As training progresses, the "width" of the Gaussian function ($\delta$) increases linearly. This causes the attention weights to gradually flatten out until, by the end of training, all pixels are weighted equally, returning the model to a standard training state.

Evolution of the attention function and resulting maps over training steps, showing the transition from focusing on hard pixels to uniform weighting.

Experimental Setup

The models were trained and evaluated using standard benchmarks in the SR community:

  • Training: DIV2K (800 high-quality images).
  • Evaluation/Testing: Set5, Set14, B100, and Urban100. These datasets vary in complexity, with Urban100 being particularly challenging due to its repetitive structures and fine lines.

Discussion: Why Pixel-Level Pacing Works

  • Pixel-wise Granularity: The core innovation is treating an image not as a single data point, but as a collection of thousands of learning tasks (pixels). By focusing on the pixels the model is currently "failing" at, the training process becomes more efficient.
  • SSIM over L1/L2: Instead of using simple pixel-wise subtraction, the authors use SSIM to build their similarity maps. SSIM is perception-based and considers neighboring pixels, making the attention map more robust to noise.
  • The "Decaying" Strategy: The strategy is self-extinguishing. By gradually reducing the influence of the attention map, the authors ensure that the model eventually learns to reconstruct the entire image accurately, not just the high-contrast edges.

Performance Evaluation

The PSPL strategy demonstrated significant improvements in both speed and accuracy:

  • Faster Convergence: In experiments with a lightweight EDSR model, the version using PSPL reached a PSNR of 34.4 dB in just 70 epochs, whereas the baseline model took 125 epochs to reach the same milestone.
  • Better Final Performance: Across almost all tested architectures (SRCNN, VDSR, DRRN, LapSRN, EDSR) and datasets, PSPL led to higher PSNR and SSIM scores compared to standard training.
  • Visual Quality: Qualitative results show that PSPL-trained models produce sharper edges and fewer artifacts in complex textures (like building facades in Urban100) compared to their baseline counterparts.

Conclusion and Future Directions

While PSPL improved the SSIM on the Urban100 dataset for the EDSR model, the PSNR was slightly lower than the baseline in some instances. This suggests that while the structures are more accurate, the absolute pixel values might vary slightly. Additionally, while PSPL reduces the number of epochs needed, calculating SSIM maps and attention maps adds a small amount of computational overhead per iteration.

The authors propose extending this pixel-level weighting strategy to other "pixel-wise regression" tasks, such as depth estimation or image deblurring, where spatial difficulty is non-uniform.

Summary updated

BibTeX
@inproceedings{lin2020pixel,
  author = {Wei Lin and Junyu Gao and Qi Wang and Xuelong Li},
  title = {{Pixel-Level Self-Paced Learning For Super-Resolution}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2020},
  doi = {10.1109/icassp40776.2020.9054293},
}
Figure