Introduction
Capturing digital screens with a camera often results in distracting, wavy rainbow patterns known as moiré. These artifacts arise from frequency aliasing between the camera’s color filter array (CFA) and the screen’s sub-pixel grid. While image-based demoiréing has seen significant progress, video demoiréing remains a formidable challenge due to spatial variation, channel dependence in RAW data, and extreme temporal instability. Because moiré patterns are aliasing artifacts, even minor camera movements cause the patterns to flicker or shift violently between frames, often leading to global color distortions.
MoCHA-former (Moiré-Conditioned Hybrid Adaptive Transformer) is a lightweight, state-of-the-art framework designed to address these issues. By explicitly disentangling moiré artifacts from clean content and leveraging frequency-domain cues, MoCHA-former outperforms existing models in both RAW and sRGB domains while maintaining a significantly smaller parameter footprint.
The MoCHA-former Architecture
MoCHA-former adopts a two-stage approach to restoration, focusing first on spatial disentanglement and subsequently on temporal consistency.
1. Decoupled Moiré Adaptive Demoiréing (DMAD)
The DMAD module is designed to separate moiré artifacts from the underlying scene content. Rather than treating the task as a simple image-to-image translation, DMAD uses two parallel branches:
- Moiré and Content Branches: The Moiré Disentanglement Block (MDB) and Moiré Prediction Block (MPB) predict the clean content and the moiré pattern independently.
- Decoupling Distillation Block (DDB): To ensure true separation, the DDB employs specialized losses—including a moiré prediction loss based on self-similarity—to prevent the model from passing identical features through both branches.
- Moiré-adaptive Cross-attention (MCB): This component uses the predicted moiré pattern as a spatial map, guiding the restoration branch to focus its computational energy on affected areas while preserving detail in clean regions.
2. Spatio-Temporal Adaptive Demoiréing (STAD)
The STAD module addresses the temporal complexities of video. Traditional methods often rely on explicit alignment (like optical flow), which frequently fails because moiré patterns do not follow the physical motion of objects in a scene. MoCHA-former instead utilizes implicit alignment via a modified Video Swin Transformer with 3D window attention.
Furthermore, STAD incorporates a Frequency-domain Refinement strategy. The authors observed that in the Fourier domain, the amplitude component encodes the structural grid of the moiré, while the phase component encodes color shifts and smoothing. Using Fourier Channel Attention (FCA) and a Window-based Frequency-adaptive Refinement Block (WFB), the model surgically removes artifacts by processing these components separately, capturing large-scale moiré structures through dilated convolutions on the amplitude.
Experimental Results and Insights
Performance Benchmarks
MoCHA-former was evaluated on the RawVDemoiré and VDemoiré datasets. The RawVDemoiré dataset is particularly significant as it allows the model to exploit channel-specific statistics in the RAW domain before the Image Signal Processor (ISP) smears artifacts across the sRGB space.
- Accuracy: On RawVDemoiré, MoCHA-former achieved a PSNR of 29.707 dB, a substantial +1.00 dB improvement over the previous state-of-the-art.
- Efficiency: The model is remarkably efficient, utilizing only 3.64 million parameters. This represents a 45% reduction in size compared to the previous leading model (6.59 million parameters).
- Visual Quality: Qualitative analysis demonstrates that MoCHA-former is superior at removing "stripe" moiré and restoring accurate color tones, whereas competing models like RRID or DTNet often leave localized color distortions.
Key Takeaways
- Implicit Alignment is Superior: For aliasing artifacts like moiré, implicit alignment via Transformers is more robust than explicit motion-based alignment.
- RAW Domain Advantages: Performing demoiréing early in the ISP pipeline preserves color correlation and allows for more effective artifact removal.
- Fourier Decomposition: Separating amplitude and phase allows the model to target the structural "grid" of the moiré without damaging the underlying scene structure.
Conclusion and Future Work
MoCHA-former represents a significant step forward in video restoration, combining high-fidelity results with a lightweight architecture suitable for resource-constrained environments. While the model sets new benchmarks, future research may focus on reducing the computational overhead of FFT/IFFT operations for mobile deployment and simplifying the current two-stage training process into a single-stage end-to-end pipeline.
Summary updated