A Preliminary Exploration Towards General Image Restoration

Xiangtao Kong, Jinjin Gu, Yihao Liu, Wenlong Zhang, Xiangyu Chen, Yu Qiao, Chao DongarXiv 2024 · DOI · arXiv

Introduction

In contemporary computer vision, image restoration has traditionally been approached through specialized models designed for isolated degradations, such as denoising, deblurring, or super-resolution. While these models perform exceptionally well on "clean" synthetic problems, they often fail when confronted with the "messy reality" of real-world imagery. A typical smartphone photo taken in adverse conditions may simultaneously suffer from sensor noise, motion blur, JPEG compression artifacts, and atmospheric haze.

This paper introduces General Image Restoration (GIR), a paradigm shift toward a unified, "blind" model capable of handling complex, unknown, and combined degradations without explicit task-specific instructions. The authors propose a formal framework and a benchmarking protocol to evaluate progress in this challenging domain.

Conceptual framework of General Image Restoration (GIR) addressing infinite continuously changing tasks compared to discrete high-level vision tasks.

Methodology and Research Framework

Rather than proposing a single architecture, the study establishes a comprehensive research framework for GIR.

Problem Definition

GIR is mathematically defined as a structural risk minimization problem. The core objective is to learn the underlying distribution of natural, clear images rather than merely learning to invert specific mathematical degradation functions. This approach emphasizes the model's ability to distinguish "naturalness" from various forms of "unnaturalness."

Stochastic Degradation Pipeline

To train models for the diversity of GIR, the authors utilize a stochastic pipeline that randomly composes multiple basic degradation functions (e.g., blur, noise, resizing) in varying orders and intensities. This process generates "mixture tasks" that better approximate real-world conditions compared to standard single-task datasets.

Evaluation Protocol: AR & ER

Standard metrics like Peak Signal-to-Noise Ratio (PSNR) are often incomparable across different restoration tasks. To address this, the authors introduced two relative metrics:

  • Acceptance Ratio (AR): Measures how frequently a model outperforms a basic 3-layer CNN baseline.
  • Excellence Ratio (ER): Measures how often a general model matches or exceeds the performance of a state-of-the-art (SOTA) model specifically tuned for a single task.

Data and Insights

The study utilized high-quality synthetic data (MiO100) and diverse real-world samples. A key finding was that data quality outweighs quantity; using only 25% of the training data yielded results comparable to using the full set, provided the ground truth was of high quality. Interestingly, the inclusion of ImageNet data actually hindered performance due to pre-existing artifacts in its images.

Analysis of Deep Degradation Representations (DDR) revealed that successful GIR models tend to cluster different degradations less distinctly in their latent space. Models that treat noise and blur as variations of the same "unnaturalness" generalize better than those that categorize them strictly.

Key Results and the "Excellence Gap"

Benchmarking several prominent architectures (including RRDB, SwinIR, and Restormer) showed that while GIR-Restormer achieved the highest Acceptance Ratio (0.47), the Excellence Ratio remained strikingly low, with no model exceeding 0.03. This indicates that while generalist models are becoming "acceptable," they still lag significantly behind specialized models on their specific domains.

Comparison of specialized restoration models on mixture tasks, illustrating the performance drop when handling combined degradations.

Qualitatively, however, GIR models often produced visually cleaner results on real-world "blind" images. Unlike task-specific chains (e.g., deblurring followed by denoising), which often accumulate unnatural artifacts, the unified GIR approach maintains better visual consistency.

Conclusion and Future Work

This preliminary exploration highlights the inherent biases in current Transformer-based architectures, such as Restormer’s difficulty with image resizing. Future work must focus on closing the "Excellence Gap" and developing backbones specifically optimized for the diversity of GIR. The ultimate goal is to move toward "intelligent" restoration, where models leverage semantic content understanding to reconstruct missing details in highly degraded images.

Summary updated

Abstract

Despite the tremendous success of deep models in various individual image restoration tasks, there are at least two major technical challenges preventing these works from being applied to real-world usages: (1) the lack of generalization ability and (2) the complex and unknown degradations in real-world scenarios. Existing deep models, tailored for specific individual image restoration tasks, often fall short in effectively addressing these challenges. In this paper, we present a new problem called general image restoration (GIR) which aims to address these challenges within a unified model. GIR covers most individual image restoration tasks (\eg, image denoising, deblurring, deraining and super-resolution) and their combinations for general purposes. This paper proceeds to delineate the essential aspects of GIR, including problem definition and the overarching significance of generalization performance. Moreover, the establishment of new datasets and a thorough evaluation framework for GIR models is discussed. We conduct a comprehensive evaluation of existing approaches for tackling the GIR challenge, illuminating their strengths and pragmatic challenges. By analyzing these approaches, we not only underscore the effectiveness of GIR but also highlight the difficulties in its practical implementation. At last, we also try to understand and interpret these models' behaviors to inspire the future direction. Our work can open up new valuable research directions and contribute to the research of general vision.
BibTeX
@misc{kong2024preliminary,
  author = {Xiangtao Kong and Jinjin Gu and Yihao Liu and Wenlong Zhang and Xiangyu Chen and Yu Qiao and Chao Dong},
  title = {{A Preliminary Exploration Towards General Image Restoration}},
  year = {2024},
  eprint = {2408.15143},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.15143},
}
Figure