Global-Local Stepwise Generative Network for Ultra High-Resolution Image Restoration

Xin Feng, Haobo Ji, Wenjie Pei, Fanglin Chen, David Zhang, Guangming LuarXiv 2022 · OpenReview

TL;DR

This paper introduces GLSGN, a novel network for restoring ultra high-resolution (4K) images by combining multiple local and one global restoration pathways. It also presents UHR4K, the first large-scale 4K dataset for reflection and rain streak removal, demonstrating superior performance across various restoration tasks.

What Problem Does This Paper Address?

Restoring images at ultra high-resolution (like 4K) is significantly more challenging than standard resolutions. This is due to a dramatic increase in computational complexity and memory requirements, making existing methods impractical. Furthermore, high-resolution images often contain more intricate noise patterns. A critical bottleneck has also been the lack of suitable large-scale datasets for training and evaluating models on such high resolutions, particularly for tasks like reflection and rain streak removal.

Methods

The core of the proposed solution is the Global-Local Stepwise Generative Network (GLSGN). This network employs a unique strategy with four parallel pathways: three "local" pathways and one "global" pathway.

  • Local Pathways: These pathways operate on image patches. They progressively restore the image at decreasing resolutions (e.g., 4K -> 2K -> 1/16th resolution), allowing for fine-grained detail restoration.
  • Global Pathway: This pathway processes a downscaled version of the entire image. Its role is to capture global context, such as semantic information and overall noise patterns, providing crucial guidance to the local pathways.

To ensure these pathways work harmoniously, GLSGN enforces four types of inter-pathway consistency:

  1. Low-level Content Consistency: Achieved by feeding decoded features from previous pathways and the global pathway into the current pathway.
  2. Perceptual Attention Consistency: A novel mechanism (PAC) that aligns the spatial attention maps across pathways, ensuring that noisy regions receive appropriate focus.
  3. Restoring Intensity Consistency: Utilizes a "patch normalization" technique to regularize the restoration intensity across adjacent patches, preventing visual artifacts.
  4. High-level Semantics Consistency: Employs a Laplacian Pyramid to synthesize the final high-resolution image, ensuring semantic coherence.

Datasets

The paper introduces UHR4K, a new benchmark dataset specifically designed for ultra high-resolution image restoration. It comprises 4,670 real-world and synthetic images and is divided into three subsets:

  • UHR4K-Real: For reflection removal.
  • UHR4K-Syn: For reflection removal.
  • UHR4K-Rain: For rain streak removal.

This dataset is notable for its 4K resolution, making it the first of its kind for these specific tasks. Existing datasets for reflection and rain removal are typically much lower resolution.

Key Insights and Ideas

  • Divide and Conquer for High-Res: The paper demonstrates that a hierarchical, multi-pathway approach (combining global context with progressively refined local patch restoration) is highly effective for tackling the challenges of ultra high-resolution image restoration.
  • Inter-Pathway Consistency is Crucial: Simply having multiple pathways isn't enough. The explicit modeling of consistency across different aspects (content, attention, intensity, semantics) is key to achieving seamless and high-quality results.
  • Attention Alignment Matters: The proposed Perceptual Attention Consistency (PAC) mechanism highlights that aligning how different restoration pathways "look" at an image is vital for effectively handling noise and details across scales.
  • The Need for High-Res Benchmarks: The creation of the UHR4K dataset underscores the significant gap in high-resolution data for critical image restoration tasks, which has hindered progress in this area.

Key Results

The proposed GLSGN model consistently outperforms state-of-the-art methods across three tasks: image reflection removal, image rain streak removal, and image dehazing, particularly on the new UHR4K dataset.

  • Reflection Removal: GLSGN achieves a PSNR of 24.35 dB and SSIM of 0.841 on UHR4K-Real, significantly outperforming other methods. Human evaluation also showed GLSGN winning 75.2% of votes.
  • Rain Streak Removal: On UHR4K-Rain, GLSGN achieves a PSNR of 32.83 and SSIM of 0.938, demonstrating superior performance.
  • Dehazing: On the 4KID dataset, GLSGN achieves a PSNR of 24.42 and SSIM of 0.901, also outperforming existing methods.
  • Efficiency: GLSGN is shown to be computationally efficient, requiring fewer TFLOPS, trainable parameters, and less runtime compared to many competing methods, especially for 4K images.

Limitations and Future Work

  • Handling Multiple Degradations: While GLSGN excels at individual degradation types, the authors acknowledge that handling a mixture of multiple degradations simultaneously remains a challenge. Future work could focus on learning more generalizable noise patterns for such scenarios.
  • Efficiency Improvement: The paper suggests further refinement of the modeling structure to improve computational efficiency.
  • Dataset Expansion: While UHR4K is a significant contribution, the field could benefit from even larger and more diverse datasets for ultra high-resolution restoration.

Summary updated

Abstract

While the research on image background restoration from regular size of degraded images has achieved remarkable progress, restoring ultra high-resolution (e.g., 4K) images remains an extremely challenging task due to the explosion of computational complexity and memory usage, as well as the deficiency of annotated data. In this paper we present a novel model for ultra high-resolution image restoration, referred to as the Global-Local Stepwise Generative Network (GLSGN), which employs a stepwise restoring strategy involving four restoring pathways: three local pathways and one global pathway. The local pathways focus on conducting image restoration in a fine-grained manner over local but high-resolution image patches, while the global pathway performs image restoration coarsely on the scale-down but intact image to provide cues for the local pathways in a global view including semantics and noise patterns. To smooth the mutual collaboration between these four pathways, our GLSGN is designed to ensure the inter-pathway consistency in four aspects in terms of low-level content, perceptual attention, restoring intensity and high-level semantics, respectively. As another major contribution of this work, we also introduce the first ultra high-resolution dataset to date for both reflection removal and rain streak removal, comprising 4,670 real-world and synthetic images. Extensive experiments across three typical tasks for image background restoration, including image reflection removal, image rain streak removal and image dehazing, show that our GLSGN consistently outperforms state-of-the-art methods.
BibTeX
@misc{feng2022global,
  author = {Xin Feng and Haobo Ji and Wenjie Pei and Fanglin Chen and David Zhang and Guangming Lu},
  title = {{Global-Local Stepwise Generative Network for Ultra High-Resolution Image Restoration}},
  year = {2022},
  eprint = {2207.08808},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2207.08808},
}
Figure