Benchmarking Ultra-High-Definition Image Reflection Removal

Zhenyuan Zhang, Zhenbo Song, Kaihao Zhang, Wenhan Luo, Zhaoxin Fan, Jianfeng Lu

Abstract

Single Image Reflection Removal (SIRR) is a critical task for recovering clear transmission layers from images captured through glass. However, existing methods struggle with the computational and structural complexities of Ultra-High-Definition (UHD) 4K and 8K images. This paper introduces the first large-scale UHD reflection removal benchmarks, UHDRR4K and UHDRR8K, and proposes RRFormer, a novel transformer-based architecture. RRFormer leverages a hybrid approach, combining CNN-based local feature extraction with Swin Transformer-based global context modeling and multi-scale spatial aggregation. Our results demonstrate that RRFormer sets a new state-of-the-art on both UHD and standard-definition datasets.

Introduction

Single Image Reflection Removal (SIRR) aims to recover a clear "transmission" image from a "blended" input that contains distracting reflections. While deep learning has significantly advanced this field, most research has focused on Standard-Definition (SD) or High-Definition (HD) images. As modern smartphones increasingly capture 4K and 8K photos, existing models face two major hurdles: a lack of large-scale UHD training data and the limited receptive fields of traditional Convolutional Neural Networks (CNNs), which struggle to distinguish complex reflections across massive 8K canvases.

The RRFormer Architecture

To address the challenges of UHD reflection removal, we propose RRFormer, a network designed to balance local detail with global context. The architecture consists of three specialized modules:

  1. Preprocessing Embedding Module: This module utilizes a pre-trained VGG-19 network to extract "hypercolumn" features. These provide low-level architectural details that are concatenated with the raw input, giving the network a structural "head start."
  2. Self-attention Feature Extraction Module: The core of the model employs Residual Swin Transformer Blocks (RSTB). By using shifted windows, Swin Transformers efficiently capture global pixel interactions without the prohibitive computational cost of standard transformers at high resolutions.
  3. Multi-scale Spatial Feature Extraction Module: To maintain consistency across the entire UHD frame, a Pyramid Pooling Module (PPM) aggregates contextual information from different regions (bin sizes of 4, 8, 16, and 32), helping the model understand reflections of varying sizes.
The RRFormer architecture consists of three core modules: Preprocessing Embedding, Self-attention Feature Extraction, and Multi-scale Spatial Feature Extraction.

The model is trained using a composite loss function comprising Pixel Loss (measuring color and gradient differences), Feature Loss (ensuring perceptual similarity), and Adversarial Loss to ensure realistic outputs.

UHD Benchmarks

We introduce two landmark benchmarks for the community:

  • UHDRR4K: 2,999 training and 168 testing quadruplets at $3840 \times 2160$ resolution.
  • UHDRR8K: 1,014 training and 105 testing quadruplets at $7680 \times 4320$ resolution.

Each quadruplet includes the transmission layer, reflection layer, reflection mask, and the final blended image. These were synthesized using high-quality source images with random Gaussian blurring applied to the reflection layer to simulate real-world out-of-focus glass reflections.

Experimental Results

RRFormer was benchmarked against six state-of-the-art (SOTA) methods across multiple datasets. It consistently outperformed existing models in both quantitative metrics and qualitative visual quality.

  • UHDRR4K: RRFormer achieved a PSNR of 24.71 dB, a 0.34 dB lead over the next best method.
  • UHDRR8K: RRFormer led with a PSNR of 22.41 dB, maintaining its edge in 8K environments.
  • Generalization (CDR): On the standard-definition CDR dataset, RRFormer achieved 22.33 dB PSNR, a 1.32 dB improvement over previous leaders.

Qualitatively, RRFormer excels at recovering "true tonality," avoiding the artificial brightness or color shifts common in other models when processing complex indoor scenes or expansive sky regions.

Conclusion

RRFormer demonstrates that a hybrid approach—combining CNN-based priors with transformer-based global context—is highly effective for UHD image restoration. While synthetic data remains a limitation, the introduction of the UHDRR4K and UHDRR8K datasets provides a vital foundation for future research in high-resolution reflection removal. Future work will explore optimizing transformer architectures for the extreme memory constraints of 8K processing.

Summary updated

Abstract

Deep learning based methods have achieved significant success in the task of single image reflection removal (SIRR). However, the majority of these methods are focused on High-Definition/Standard-Definition (HD/SD) images, while ignoring higher resolution images such as Ultra-High-Definition (UHD) images. With the increasing prevalence of UHD images captured by modern devices, in this paper, we aim to address the problem of UHD SIRR. Specifically, we first synthesize two large-scale UHD datasets, UHDRR4K and UHDRR8K. The UHDRR4K dataset consists of $2,999$ and $168$ quadruplets of images for training and testing respectively, and the UHDRR8K dataset contains $1,014$ and $105$ quadruplets. To the best of our knowledge, these two datasets are the first largest-scale UHD datasets for SIRR. Then, we conduct a comprehensive evaluation of six state-of-the-art SIRR methods using the proposed datasets. Based on the results, we provide detailed discussions regarding the strengths and limitations of these methods when applied to UHD images. Finally, we present a transformer-based architecture named RRFormer for reflection removal. RRFormer comprises three modules, namely the Prepossessing Embedding Module, Self-attention Feature Extraction Module, and Multi-scale Spatial Feature Extraction Module. These modules extract hypercolumn features, global and partial attention features, and multi-scale spatial features, respectively. To ensure effective training, we utilize three terms in our loss function: pixel loss, feature loss, and adversarial loss. We demonstrate through experimental results that RRFormer achieves state-of-the-art performance on both the non-UHD dataset and our proposed UHDRR datasets. The code and datasets are publicly available at https://github.com/Liar-zzy/Benchmarking-Ultra-High-Definition-Single-Image-Reflection-Removal.
BibTeX
@misc{zhang2023benchmarking,
  author = {Zhenyuan Zhang and Zhenbo Song and Kaihao Zhang and Wenhan Luo and Zhaoxin Fan and Jianfeng Lu},
  title = {{Benchmarking Ultra-High-Definition Image Reflection Removal}},
  year = {2023},
  eprint = {2308.00265},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2308.00265},
}
Figure