RepSR: Training Efficient VGG-style Super-Resolution Networks with Structural Re-Parameterization and Batch Normalization

Xintao Wang, Chao Dong, Ying Shan

Introduction

Single Image Super-Resolution (SISR) has seen significant advancements through the use of deep convolutional neural networks (CNNs) and Transformers. However, a persistent challenge remains: the trade-off between restoration quality (measured by PSNR) and inference efficiency. While complex architectures like ResNets, DenseNets, and SwinIR achieve state-of-the-art results, their multi-branch or attention-based structures often lead to high memory consumption and reduced throughput on real-world hardware.

Plain VGG-style networks, consisting of a simple stack of 3x3 convolutions, offer the highest hardware efficiency. Yet, these models are notoriously difficult to optimize for SISR. In high-level vision tasks, Batch Normalization (BN) is the standard solution for stabilizing training, but in the SR community, BN is traditionally avoided as it is known to degrade performance and introduce visual artifacts. This paper introduces RepSR, a framework that successfully integrates BN into SR training through structural re-parameterization and a novel "FrozenBN" strategy.

The BN Dilemma in Super-Resolution

The reluctance to use BN in SR stems from a phenomenon the authors identify as "patch-level inconsistency." Unlike classification tasks where global features are prioritized, SR models rely heavily on local patch statistics. During training, BN uses mini-batch statistics to normalize features. However, in SR, a specific local patch (e.g., a smooth sky or a highly textured edge) may have statistics that deviate significantly from the global mini-batch mean and variance.

When the model switches to using running (population) statistics during inference, this discrepancy causes the normalization to "explode" or shift features inappropriately, resulting in the characteristic artifacts associated with BN in SR. RepSR addresses this by recognizing that BN is not inherently bad for SR, but rather its application during the transition from training to inference must be handled carefully.

Methodology

1. Structural Re-parameterization: The RepSR Block

To leverage the benefits of multi-branch training while maintaining the speed of a single-branch inference model, RepSR employs structural re-parameterization. During the training phase, each block consists of a multi-branch structure:

  • A clean residual path without BN to facilitate detail propagation.
  • Two expand-and-squeeze paths (comprising a 3x3 convolution followed by BN and a 1x1 convolution).

A key distinction from previous methods like ECBSR is the ordering of operations. RepSR places the 3x3 convolution before the 1x1 convolution. This design choice avoids the padding-related overhead that slowed down previous re-parameterization techniques, resulting in a training process that is approximately 50% faster. At inference time, these linear branches are mathematically fused (collapsed) into a single 3x3 convolutional layer, ensuring zero additional computational cost compared to a plain VGG network.

2. The FrozenBN Strategy

To resolve the artifact issue without losing the optimization benefits of BN, the authors propose FrozenBN. The training is divided into two stages:

  1. Initial Training: The model is trained using standard mini-batch BN statistics, allowing for rapid convergence and stable gradient flow.
  2. Fine-tuning: For the final ~10% of training iterations, the BN layers are "frozen." The model switches from using mini-batch statistics to using the accumulated population statistics (moving average). This transition eliminates the train-test inconsistency, allowing the network to adapt its weights to the exact statistics it will encounter during deployment, thereby removing artifacts.

Experimental Results

The RepSR models were trained on the DIV2K dataset and evaluated across standard benchmarks, including Set5, Set14, BSD100, and Urban100.

Performance and Throughput

RepSR demonstrates a superior trade-off between PSNR and inference throughput. On an NVIDIA A100 GPU, RepSR matches the restoration quality of complex models like RCAN and EDSR while achieving significantly higher images-per-second. Unlike earlier re-parameterization methods that were limited to "tiny" models, RepSR scales effectively, showing consistent gains as the model depth and width increase. Visual results on the Urban100 dataset specifically highlight RepSR's ability to recover sharper edges and clearer line structures compared to standard VGG or EDSR models.

Training Efficiency

By optimizing the branch structure and utilizing BN, RepSR achieves faster convergence than its predecessors. Compared to ECBSR, which relies on hand-designed filters (e.g., Sobel or Laplacian), RepSR's data-driven approach reduces training time by half while maintaining or exceeding performance levels.

Conclusion

RepSR challenges the long-standing convention that Batch Normalization is unsuitable for Super-Resolution. By combining structural re-parameterization with the FrozenBN strategy, the authors enable the training of high-performance, VGG-style networks that are both easy to optimize and exceptionally fast during inference. While challenges remain in scaling plain networks to extreme depths (e.g., >100 layers), RepSR provides a robust blueprint for developing hardware-efficient SR solutions that do not sacrifice image quality.

Summary updated

BibTeX
@inproceedings{wang2022repsr,
  author = {Xintao Wang and Chao Dong and Ying Shan},
  title = {{RepSR: Training Efficient VGG-style Super-Resolution Networks with Structural Re-Parameterization and Batch Normalization}},
  booktitle = {ACM Multimedia},
  year = {2022},
  url = {https://openreview.net/forum?id=w7HYDNa4gT_d},
}
Figure