Two-stage domain adapted training for better generalization in real-world image restoration and super-resolution

Cansu Korkmaz, A. Murat Tekalp, Zafer Dogan

Introduction

In computer vision, inverse problems such as Super-Resolution (SR) and image deblurring are increasingly solved using deep neural networks. While these models achieve high performance when trained on synthetic data—such as images downsampled using a perfect bicubic filter—they often fail when applied to real-world imagery. Real-world degradations, including camera shake, sensor noise, and lens blur, rarely follow simple mathematical formulas. Consequently, end-to-end networks often overfit to synthetic distortions, producing significant artifacts or failing to restore details in "blind" restoration scenarios where the exact nature of the image damage is unknown.

Methodology: Two-Stage Domain Adapted Training

To address the generalization gap, this paper introduces a Two-Stage Domain Adapted Training strategy. Rather than attempting to map a "messy" real-world input directly to a clean output, the process is decoupled into two distinct phases: Domain Mapping and Task Execution.

Overview of the proposed two-stage domain adapted training strategy, illustrating the various paths from unknown low-resolution images to the final super-resolution (SR) output through intermediate mapped domains.

1. Stage 1: Domain Mapping

In the first stage, the input image with unknown degradation is passed through a mapper network—specifically a Residual Channel Attention Network (RCAN). The objective is to transform the input into a known, standardized intermediate domain. For SR tasks, this means converting an unknown low-resolution (LR) image into one that mimics the characteristics of a standard bicubic downsampling.

2. Stage 2: Task Execution

A second network then takes this standardized intermediate image to perform the final task, such as 4x super-resolution or deblurring. By standardizing the input domain, the task-specific network can be trained on high-quality synthetic data and still perform effectively on real-world inputs.

Key Innovations

  • Intermediate Resolution (The "Stepping Stone"): For 4x SR, the authors discovered that mapping the input to a 2x resolution intermediate domain, rather than keeping it at the original input resolution, yielded the best results. This acts as a stepping stone, allowing the network to correct artifacts and increase resolution in manageable stages.
  • L1 Loss for Domain Adaptation: Unlike previous approaches that utilized Generative Adversarial Networks (GANs)—which can introduce hallucinated textures—this method employs a simple L1 loss with RCAN, resulting in cleaner and more faithful reconstructions.

Experiments and Results

The researchers evaluated their framework using the DIV2K "Unknown" benchmark and the RealSR dataset, which contains authentic image pairs captured with varying focal lengths on Canon and Nikon cameras.

Visual comparison of super-resolution results on a real-world lattice structure. The proposed two-stage method (f) produces sharper edges and fewer ringing artifacts compared to standard direct training approaches.

Quantitative Performance

On the DIV2K "Unknown" 4x SR task, the two-stage "Mapping x2" approach achieved a PSNR of 28.922 dB, outperforming direct RCAN training (28.819 dB). On the RealSR Nikon dataset, the method achieved a state-of-the-art PSNR of 27.107 dB and an SSIM of 0.775, demonstrating superior handling of actual camera-captured degradations.

Qualitative Analysis

As shown in the visual comparisons, the two-stage approach produces significantly sharper edges and fewer "ringing" artifacts than direct end-to-end models. The decoupling of blur standardization from detail reconstruction allows the model to handle unknown blurs with remarkable precision, even outperforming direct training by up to 19dB PSNR in specific deblurring tests.

Conclusion

The proposed two-stage domain adapted training strategy effectively mitigates the overfitting problem in real-world image restoration. By standardizing unknown degradations into a known intermediate domain, the framework achieves superior generalization and visual quality. Future work will explore the optimization of task-specific intermediate domains and methods to reduce the computational overhead of the dual-network architecture.

Summary updated

Abstract

It is well-known that in inverse problems, end-to-end trained networks overfit the degradation model seen in the training set, i.e., they do not generalize to other types of degradations well. Recently, an approach to first map images downsampled by unknown filters to bicubicly downsampled look-alike images was proposed to successfully super-resolve such images. In this paper, we show that any inverse problem can be formulated by first mapping the input degraded images to an intermediate domain, and then training a second network to form output images from these intermediate images. Furthermore, the best intermediate domain may vary according to the task. Our experimental results demonstrate that this two-stage domain-adapted training strategy does not only achieve better results on a given class of unknown degradations but can also generalize to other unseen classes of degradations better.
BibTeX
@inproceedings{korkmaz2021two,
  author = {Cansu Korkmaz and A. Murat Tekalp and Zafer Dogan},
  title = {{Two-stage domain adapted training for better generalization in real-world image restoration and super-resolution}},
  booktitle = {ICIP},
  year = {2021},
  doi = {10.1109/icip42928.2021.9506380},
}
Figure