Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data

Xintao Wang, Liangbin Xie, Chao Dong, Ying Shan

Introduction

Super-Resolution (SR) aims to reconstruct high-resolution (HR) images from low-resolution (LR) inputs. While deep learning has significantly advanced this field, most existing models are trained on "clean" synthetic data—typically created using simple bicubic downsampling. This creates a significant performance gap when these models encounter real-world images, which often suffer from complex, overlapping degradations like camera noise, motion blur, and aggressive JPEG compression. Real-ESRGAN addresses this "blind" SR challenge by introducing a sophisticated data synthesis pipeline that mimics the messy reality of digital imagery.

Methodology

The primary contribution of Real-ESRGAN is its approach to data synthesis rather than a radical change in the generator architecture. The model utilizes the established ESRGAN framework but trains it on a novel "high-order" degradation model.

High-Order Degradation Modeling

Traditional SR models use a first-order degradation process (e.g., Blur → Resize → Noise → JPEG). Real-ESRGAN extends this to a second-order process, applying the degradation pipeline twice. This multi-step approach better simulates the lifecycle of real-world images, such as a photo taken on a smartphone, compressed for storage, and then resized for social media.

The high-order degradation modeling process used to synthesize realistic training data.

Sinc Filters and Ringing Artifacts

To combat common artifacts like "ringing" (ghostly lines near edges) and "overshoot" (bright halos), the authors integrate sinc filters into the synthesis pipeline. These filters simulate the effects of sharpening and limited bandwidth, allowing the model to learn how to suppress these specific visual distortions.

U-Net Discriminator

In the GAN training phase, the standard VGG-style discriminator is replaced with a U-Net architecture. Unlike global discriminators that provide a single "real/fake" score for an entire image, the U-Net discriminator provides per-pixel feedback. This allows the generator to focus on restoring intricate local textures and details, which is crucial for high-fidelity restoration.

Experimental Results

Real-ESRGAN was trained on high-quality datasets (DIV2K, Flickr2K) using purely synthetic degradations. Despite the lack of real-world training pairs, the model demonstrates exceptional generalization across various real-world test sets.

Qualitative Analysis

As shown in the comparative results, Real-ESRGAN significantly outperforms its predecessors. It effectively removes the aggressive noise and artifacts seen in Bicubic and ESRGAN outputs while maintaining sharper edges and more realistic textures. In particular, it excels at restoring text (as seen in the "Cheesecake Factory" sign) and complex natural structures like tree branches, where other methods often produce blurry or over-smoothed results.

Qualitative comparison of Real-ESRGAN against other super-resolution methods across different image domains.

Quantitative Performance

Using the Natural Image Quality Evaluator (NIQE) metric, Real-ESRGAN consistently achieves lower (better) scores compared to state-of-the-art methods like BSRGAN and the original ESRGAN. For instance, on the RealSR-Canon dataset, Real-ESRGAN achieved a NIQE score of 4.53, a substantial improvement over ESRGAN's 6.77.

Discussion and Limitations

While Real-ESRGAN represents a major step forward in blind SR, it is not without limitations. The model can occasionally struggle with aliasing artifacts (e.g., jagged lines on buildings) and may sometimes hallucinate textures that were not present in the original scene—a common side effect of GAN-based training. Furthermore, if an input image contains degradations significantly outside the scope of the second-order model, the restoration quality may degrade. Future research will likely explore even higher-order modeling and more robust ways to distinguish between signal and noise.

Summary updated

BibTeX
@inproceedings{wang2021real,
  author = {Xintao Wang and Liangbin Xie and Chao Dong and Ying Shan},
  title = {{Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data}},
  booktitle = {ICCVW},
  year = {2021},
  url = {https://openreview.net/forum?id=udfrio5_CW},
}
Figure