Recaptured Raw Screen Image and Video Demoiréing via Channel and Spatial Modulations

Huanjing Yue, Yijia Cheng, Xin Liu, Jingyu Yang

Abstract: Capturing screen contents by smartphone cameras has become a common way for information sharing. However, these images and videos are often degraded by moiré patterns, which are caused by frequency aliasing between the camera filter array and digital display grids. We observe that the moiré patterns in raw domain is simpler than those in sRGB domain, and the moiré patterns in raw color channels have different properties. Therefore, we propose an image and video demoiréing network tailored for raw inputs. We introduce a color-separated feature branch, and it is fused with the traditional feature-mixed branch via channel and spatial modulations. Specifically, the channel modulation utilizes modulated color-separated features to enhance the color-mixed features. The spatial modulation utilizes the feature with large receptive field to modulate the feature with small receptive field. In addition, we build the first well-aligned raw video demoiréing (RawVDemoiré) dataset and propose an efficient temporal alignment method by inserting alternating patterns. Experiments demonstrate that our method achieves state-of-the-art performance for both image and video demoiréing.

Introduction

When you point a smartphone at a computer monitor or TV, the resulting image often contains wavy, colored stripes known as moiré patterns. This phenomenon is caused by frequency aliasing—the grid of pixels on the digital display interferes with the grid of color filters on the camera sensor. While most existing solutions attempt to fix this in the standard sRGB domain (the final processed image), this paper demonstrates that processing the data in the RAW sensor domain is significantly more effective.

The Advantage of the RAW Domain

By the time an image reaches the sRGB stage, the camera's Image Signal Processor (ISP) has already performed demosaicing. This process interpolates missing color data and effectively "smears" the moiré artifacts across all color channels, making them much harder to untangle.

In the RAW domain, however, moiré patterns are often confined to specific color channels. For instance, the green channel typically has a higher sampling rate and may remain relatively clean compared to the red or blue channels.

Comparison of moiré patterns in sRGB vs. individual RAW color channels, highlighting the "clean channel" hypothesis.

Methods: Channel and Spatial Modulation (CSMod)

The proposed CSMod network utilizes a dual-branch architecture to exploit these RAW domain characteristics:

  1. Channel Modulation: Instead of standard convolutions that mix color channels immediately, the network uses Group Convolutions (GConv) to process the R, G1, G2, and B channels separately. These color-separated features are then used to modulate (scale and enhance) traditional color-mixed features, allowing the network to use information from cleaner channels to fix degraded ones.
  2. Spatial Modulation: Moiré patterns often have a large spatial period. To address this, the authors use a large-kernel (11x11) depth-wise convolution to capture a wide receptive field. This global context is used to modulate features with smaller receptive fields, helping the network identify and eliminate the periodic patterns.
  3. Video Alignment: For video sequences, a Pyramid, Cascading, and Deformable (PCD) alignment module is used to handle motion between frames before they are fused for restoration.

The RawVDemoiré Dataset

A significant contribution of this work is RawVDemoiré, the first real-world dataset for raw video demoiréing. A major challenge in creating such a dataset is "temporal confusion"—the mismatch between screen refresh rates and camera frame rates, which can lead to captured frames that are a mix of two different source frames.

The data collection pipeline showing the use of alternating "slash" patterns to ensure perfect temporal alignment.

To solve this, the authors inserted alternating "slash" and "backslash" patterns in the borders of the source video. If a captured frame shows a "cross" pattern, it is identified as a "confused" frame and discarded, ensuring the final dataset is perfectly aligned with the ground truth.

Key Results

The proposed method outperforms previous state-of-the-art (SOTA) models across both image and video tasks:

  • Image Demoiréing: Achieved a 27.26 dB PSNR, outperforming previous best models while using fewer parameters.
  • Video Demoiréing: Achieved 28.71 dB PSNR, a significant ~1.0 dB improvement over the previous SOTA.

Qualitative comparisons show that while other models often leave behind residual color tints or "ghost" stripes, the CSMod approach produces much cleaner textures and more accurate color tones. This research highlights the importance of RAW domain processing for high-fidelity image restoration in the age of ubiquitous screen sharing.

Summary updated

Abstract

Capturing screen contents by smartphone cameras has become a common way for information sharing. However, these images and videos are often degraded by moir\'e patterns, which are caused by frequency aliasing between the camera filter array and digital display grids. We observe that the moir\'e patterns in raw domain is simpler than those in sRGB domain, and the moir\'e patterns in raw color channels have different properties. Therefore, we propose an image and video demoir\'eing network tailored for raw inputs. We introduce a color-separated feature branch, and it is fused with the traditional feature-mixed branch via channel and spatial modulations. Specifically, the channel modulation utilizes modulated color-separated features to enhance the color-mixed features. The spatial modulation utilizes the feature with large receptive field to modulate the feature with small receptive field. In addition, we build the first well-aligned raw video demoir\'eing (RawVDemoir\'e) dataset and propose an efficient temporal alignment method by inserting alternating patterns. Experiments demonstrate that our method achieves state-of-the-art performance for both image and video demori\'eing. We have released the code and dataset in https://github.com/tju-chengyijia/VD_raw.
BibTeX
@inproceedings{yue2023recaptured,
  author = {Huanjing Yue and Yijia Cheng and Xin Liu and Jingyu Yang},
  title = {{Recaptured Raw Screen Image and Video Demoiréing via Channel and Spatial Modulations}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=EkcO9tHm6S},
}
Figure