Introduction
Video denoising is a critical task in computer vision, particularly for the restoration of digitized historic films. These videos often contain "analog noise"—spatially correlated grain from film stock and scanning. While transformer-based models like VRT offer high quality, their quadratic computational complexity makes them impractical for 4K cinematic content. Furthermore, reliance on explicit motion compensation can lead to artifacts. This paper introduces ASwin, a lightweight, "blind" denoising network that avoids these pitfalls.
Methods
The proposed architecture utilizes a residual UNet structure enhanced by two key innovations:
- Aggregated Shifted Window (ASwin) Attention: To manage computational costs, ASwin employs local windows similar to Swin Transformers but extends them temporally. It aggregates intra-frame self-attention and extra-frame cross-attention using a shared projection matrix and a residual aggregation scheme, maintaining a constant memory footprint regardless of the number of neighboring frames.
- Axial-Coronal-Sagittal (ACS) Convolutions: Instead of resource-intensive 3D convolutions, the model uses ACS convolutions. By splitting a 3D kernel into three 2D components along different axes, it captures temporal information with the parameter efficiency of 2D operations.
To train for real-world scenarios, the authors developed a degradation pipeline simulating complex analog grain through a combination of Gaussian, Poisson, and speckle noise, plus JPEG compression.
Key Results
The ASwin model demonstrates a significant leap in efficiency without sacrificing substantial quality.
- Efficiency: The model achieves a 21.2x speedup over the VRT model. On a 960x540 frame, it runs in 0.37s, compared to 7.86s for VRT and 24.64s for PaCNet.
- Quantitative Performance: On synthetic benchmarks (DAVIS, $\sigma=10$), ASwin reached a PSNR of 40.15 dB, closely trailing VRT's 40.82 dB.
- Real-world Quality: In non-reference assessments (MUSIQ), ASwin scored 38.16, outperforming commercial tools like NeatVideo (33.14).
A blind user study further validated these results, with 82% of participants preferring the ASwin method over other academic state-of-the-art models.
Limitations
Despite its strengths, the model faces challenges. Temporal downsampling in the UNet bottleneck did not yield performance gains, and the model may struggle with extremely fast motion where shifted windows fail to overlap sufficiently across frames. Future work will explore the optimal balance between spatial window size and training stability.
Summary updated