Unpaired Screen-Shot Image Demoiréing With Cyclic Moiré Learning

Hyunkook Park, Vien Gia An, Hanul Kim, Yeong Jun Koh, Chul LeeIEEE Access 2022 · DOI

Introduction

Capturing images of digital displays often results in the appearance of disruptive moiré artifacts—colorful, wavy patterns caused by frequency aliasing between the camera's sensor and the screen's subpixel grid. While deep learning models have shown promise in removing these artifacts, most existing solutions rely on supervised learning with perfectly aligned "clean" and "moiré" image pairs. Collecting such datasets is labor-intensive and difficult to scale. This paper proposes a novel "cyclic moiré learning" framework designed for unpaired screen-shot image demoiréing, enabling effective training using separate sets of clean and moiré images.

Examples of various moiré artifacts across different scenes, including screen-shots and natural textures.

Proposed Methodology

The core of the proposed approach is a dual-network system that learns the bidirectional mapping between clean and moiré domains without requiring paired data.

The Moiréing Network ($G_M$)

To facilitate demoiréing without pairs, the system first learns to synthesize realistic moiré. The authors identify that moiré degradation involves both local patterns and global intensity shifts. Consequently, $G_M$ is decoupled into two modules:

  • Intensity Degradation Module: This module utilizes an image histogram and a compact neural network to estimate a global brightness scaling factor, mimicking the typical darkening observed in screen captures. Using a 256-dimensional histogram provides a computationally efficient way to give the network global context.
  • Moiré Generation Module: A coarse-to-fine U-Net architecture is employed to generate multi-scale wavy patterns, ensuring the synthetic artifacts match the complexity of real-world moiré.

The Demoiréing Network ($G_D$)

The demoiréing network is trained on "pseudo-paired" data generated by $G_M$. By applying $G_D$ to the synthetic moiré images, the model has access to the original clean images as ground truth. To maintain structural integrity, the authors introduce the Advanced Sobel Loss (ASL). This loss focuses on edge maps, ensuring the network removes the "fake" edges created by moiré ripples while preserving the "real" edges of the actual objects in the photo.

Experimental Results

The framework was evaluated on three major benchmarks: LCDMoire (synthetic), TIP2018 (real-world), and MRBI (mobile screens). The model demonstrated strong generalization, performing effectively on datasets with different characteristics than those used in training.

Quantitative and Qualitative Performance

On the LCDMoire dataset, the proposed method achieved a Peak Signal-to-Noise Ratio (PSNR) of 28.00 dB, a substantial lead over the previous state-of-the-art unsupervised method, CycleGAN (26.23 dB). Visually, the model excels at restoring color vibrancy and removing artifacts without the "washing out" effect common in other unsupervised approaches. It effectively preserves the underlying digital content while eliminating the aliasing patterns.

A detailed view of moiré artifacts on a license plate, demonstrating the complex patterns the model is designed to remove.

Efficiency

Despite the complex cyclic architecture, the model maintains a reasonable inference time of approximately 0.27 seconds for a 1024x1024 image on a modern GPU, making it practical for real-world applications.

Conclusion and Future Work

This research demonstrates that decoupling moiré generation into intensity and pattern components significantly enhances the effectiveness of unpaired demoiréing. The cyclic learning framework provides a robust alternative to supervised methods, achieving state-of-the-art results. Future work may explore Transformer-based architectures to address low-frequency, large-scale moiré patterns and extend these techniques to general texture moiré in photography, such as patterns seen on striped clothing.

Summary updated

BibTeX
@article{park2022unpaired,
  author = {Hyunkook Park and Vien Gia An and Hanul Kim and Yeong Jun Koh and Chul Lee},
  title = {{Unpaired Screen-Shot Image Demoiréing With Cyclic Moiré Learning}},
  journal = {IEEE Access},
  year = {2022},
  doi = {10.1109/access.2022.3149478},
}
Figure