When you point a smartphone camera at a laptop or television screen, the preview image is often instantly ruined by a wave of shifting, rainbow-colored ripples. This phenomenon is a moiré pattern.
Moiré patterns are not random sensor noise, nor are they a defect of the camera lens. Instead, they are a physical manifestation of spatial frequency aliasing. They occur when two highly structured, repetitive grids—the subpixel layout of a digital display and the color filter array of an image sensor—overlap with a slight misalignment or scale difference.
Because modern life relies heavily on smartphones capturing digital documents, screens, and media, a major research direction has emerged to understand, model, and algorithmically remove these artifacts. This process is known as image and video demoiréing.
The Geometry of Frequency Aliasing
To understand why moiré patterns are so difficult to eliminate, we must look at how they form. A digital screen is composed of a dense grid of red, green, and blue subpixels. A camera sensor similarly samples the world through a discrete grid of pixels.
When the camera captures the screen, the high spatial frequencies of the display's subpixel grid often exceed the Nyquist frequency of the camera's sensor. According to sampling theory, when a signal contains frequencies higher than half the sampling rate, those high frequencies do not simply disappear; they "fold back" into lower frequencies. This creates a completely new, lower-frequency "beat" signal that manifests as visible, wavy bands.
As detailed by Radu Timofte et al. in their study of the AIM 2019 Challenge on Image Demoireing: Dataset and Study, these patterns are highly sensitive to the physical relationship between the camera and the screen. Changing the camera's distance, tilt, or viewing angle completely alters the shape, frequency, and color distribution of the moiré waves.
This variability makes demoiréing an incredibly ill-posed problem. As noted by Sangmin Kim et al. in the design of C3Net: Demoiréing Network Attentive in Channel, Color and Concatenation, traditional image denoising algorithms fail here. Stochastic noise is spatially uncorrelated and can be smoothed away, but moiré patterns are highly structured, deeply embedded within the image signal, and cause severe, non-local color distortions across channels.
[Digital Screen Grid] + [Camera Sensor Grid]
(High Freq) (High Freq)
\ /
\ /
v v
[Spatial Frequency Aliasing]
|
v
[Low-Frequency Moiré Pattern]
(Wavy, Chromatic Ripples)
Multi-Scale and Frequency-Domain Restoration
Because a single image can contain moiré artifacts that range from tiny, pixel-level color streaks to massive waves spanning the entire frame, a network operating at a single scale cannot effectively clean the image.
To address this, researchers have designed multi-scale architectures. The intuition is straightforward: by downsampling the image into a Gaussian or Laplacian pyramid, large-scale moiré ripples compress into localized features that a convolutional network can easily identify and neutralize. Concurrently, finer scales preserve the sharp edges of text and textures.
For instance, the framework proposed by Yizhou Yu et al. in Moiré Photo Restoration Using Multiresolution Convolutional Neural Networks processes images across multiple frequency bands to isolate these varying scales. Similarly, the authors of MMDM: Multi-frame and Multi-scale for Image Demoiréing combine multi-scale feature encoding with multi-frame burst alignment to separate the static scene content from the shifting interference patterns.
However, spatial downsampling risks destroying legitimate high-frequency details, such as text or fine background textures. To circumvent this trade-off, recent architectures operate directly in the frequency domain. As Vien Gia An et al. demonstrate in Dual-domain Deep Convolutional Neural Networks for Image Demoireing, processing images in both the spatial and frequency domains allows networks to distinguish between legitimate scene textures and aliasing artifacts.
By transforming the image into the wavelet domain, as explored by Xiaotong Luo et al. in Deep Wavelet Network with Domain Adaptation for Single Image Demoireing, or by using a dual-branch network like the one proposed by Bin He et al. in FHDe2Net: Full High Definition Demoireing Network, models can surgically remove periodic frequency spikes while keeping the underlying image details sharp and unblurred.
Managing Temporal Dynamics in Video
Demoiréing becomes significantly more complex when moving from static images to video sequences. When a camera moves relative to a screen, the geometric alignment between the two grids changes continuously. This causes the moiré patterns to move dynamically across the video frames, creating a highly disruptive, shimmering flicker.
If you apply an image-based demoiréing model to each frame of a video independently, the lack of temporal awareness causes the remaining artifacts to change shape from frame to frame, worsening the perceived flicker. Traditional video restoration often relies on optical flow to align frames, but optical flow algorithms easily break down when tracking images covered by shifting, non-rigid moiré ripples.
To solve this, Peng Dai et al. introduced a relation-based temporal consistency framework in Video Demoiréing with Relation-Based Temporal Consistency. Instead of trying to calculate pixel-to-pixel motion vectors through the artifacts, the authors use a relation-based loss that enforces consistency in the feature space across frames.
By looking at how features correlate across time rather than tracking exact pixel coordinates, the network can maintain a stable representation of the clean, latent display content beneath the moving ripples. This concept is expanded by Yuhui Quan et al. in Deep Video Demoiréing via Compact Invertible Dyadic Decomposition, which uses invertible neural networks to cleanly decompose video sequences into latent clean frames and dynamic moiré components without losing fine-grained details.
Real-World Adaptation and Downstream Impacts
A major hurdle in training deep learning models for demoiréing is data collection. It is nearly impossible to capture perfectly aligned pairs of "moiré" and "clean" photos of the same screen in the real world, as any slight camera movement alters the artifact.
While synthetic datasets exist, models trained purely on synthetic data often fail when deployed on real smartphones due to the complex, unpredictable nature of physical display-camera interactions. To bridge this gap, Hyunkook Park et al. proposed Unpaired Screen-Shot Image Demoiréing With Cyclic Moiré Learning. By using a cyclic learning framework, the authors trained two competing networks: one that learns to synthesize realistic moiré patterns onto clean images, and another that learns to strip them away. This allows the model to learn from unpaired real-world photographs, drastically improving generalization.
Understanding and mitigating moiré interference is critical because these artifacts do more than just degrade visual quality—they actively break downstream computer vision systems. For example, Razaib Tariq et al. demonstrated in Exploring the Impact of Moire Pattern on Deepfake Detectors that when a deepfake video is re-filmed from a screen, the resulting moiré patterns act as a natural spoofing attack. The high-frequency interference completely obliterates the subtle generative traces that neural networks use to identify manipulated content, causing detector accuracy to plummet.
Conversely, this distortion can be harnessed for security. In A Screen-Shooting Resilient Document Image Watermarking Scheme using Deep Neural Network, Sulong Ge et al. intentionally modeled moiré and screen distortions during the training of a watermarking network. By anticipating how the display-camera interaction will alter the image frequencies, the authors created invisible digital watermarks that survive the "analog hole" of smartphone photography, ensuring secure document tracking despite severe aliasing artifacts.
Key Foundational Work
A new random color filter array with good spectral properties (2009) — contributed 3 concept elements to this direction.
A New Computational Image Sensor with Programmable Spatially Variant Multiresolution Readout Capability (2000) — contributed 1 concept elements to this direction.
A projector-camera setup for geometry-invariant frequency demultiplexing (2009) — contributed 1 concept elements to this direction.