Decoupling Degradations with Recurrent Network for Video Restoration in Under-Display Camera

Chengxu Liu, Xuan Wang, Yuanting Fan, Shuai Li, Xueming Qian

Abstract

Under-display camera (UDC) systems enable full-screen displays by mounting the lens beneath the screen. However, the OLED pixel array causes significant light diffraction and attenuation, leading to complex degradations like flare and haze. We present D$^2$RNet, a novel video restoration network that decouples these degradations. By employing Decoupling Attention Modules (DAM), the network separates flare and haze components to apply targeted temporal processing. Our approach achieves state-of-the-art performance on a new large-scale UDC video benchmark, VidUDC33K.

The UDC Challenge: Flare and Haze

The primary hurdle for UDC systems is the OLED display itself. The pixel grid acts as a diffraction grating, scattering light before it reaches the sensor. This results in two distinct types of degradation:

  1. Flare: Intense light sources produce saturated streaks and "rainbow" artifacts.
  2. Haze: Darker areas suffer from a loss of contrast and blurry textures.
Physical model of UDC degradation and the PSF-based video generation process.

In video, these effects are dynamic. As the camera or objects move, the Point Spread Function (PSF) shifts, making temporal consistency difficult to maintain with standard image restoration techniques. To address this, the authors modeled the dynamic PSF to create realistic training data.

Methodology: D$^2$RNet and the DAM

The core of D$^2$RNet is the Decoupling Attention Module (DAM). Unlike previous methods that treat all degradations uniformly, DAM uses a soft mask generation function to partition frames into flare and haze regions based on light intensity.

Detailed architecture of the Decoupling Attention Module (DAM) showing separate paths for haze and flare removal.

Targeted Temporal Processing

  • Long-Term Flare Removal: Because flare often saturates pixels (destroying local content), the network uses long-term features to retrieve information from distant frames where the light source might not have been present or was in a different position.
  • Short-Term Haze Removal: Haze is primarily a texture blur issue. The network leverages short-term features from adjacent frames to sharpen details through precise temporal alignment.

The overall architecture is a multi-scale bi-directional recurrent network, allowing it to handle objects changing size and distance while maintaining high efficiency.

VidUDC33K: A New Benchmark

To train and evaluate D$^2$RNet, the authors introduced VidUDC33K, the first large-scale UDC video dataset. It consists of 677 high-quality HDR videos degraded using a realistic PSF measured from a commercial ZTE Axon 20 smartphone. This ensures the model learns to handle the specific optical characteristics of real-world UDC hardware.

Results and Performance

D$^2$RNet sets a new standard for UDC video restoration:

  • Quantitative Superiority: It achieves 31.91 dB PSNR on VidUDC33K, a 1.02 dB improvement over the previous state-of-the-art (RVRT).
  • Efficiency: With only 5.76M parameters and a runtime of 0.44s per frame, it is significantly more efficient than heavy Transformer-based models like VRT, making it viable for mobile deployment.
  • Qualitative Quality: Visual results demonstrate superior removal of diffraction artifacts and better restoration of shadow textures compared to general-purpose video restoration models.

Conclusion

By "decoupling" the complex optical degradations of UDC systems into manageable components, D$^2$RNet provides a targeted and efficient solution for full-screen camera restoration. This divide-and-conquer strategy not only improves visual quality but also ensures the temporal consistency required for high-quality video.

Summary updated

Abstract

Under-display camera (UDC) systems are the foundation of full-screen display devices in which the lens mounts under the display. The pixel array of light-emitting diodes used for display diffracts and attenuates incident light, causing various degradations as the light intensity changes. Unlike general video restoration which recovers video by treating different degradation factors equally, video restoration for UDC systems is more challenging that concerns removing diverse degradation over time while preserving temporal consistency. In this paper, we introduce a novel video restoration network, called D$^2$RNet, specifically designed for UDC systems. It employs a set of Decoupling Attention Modules (DAM) that effectively separate the various video degradation factors. More specifically, a soft mask generation function is proposed to formulate each frame into flare and haze based on the diffraction arising from incident light of different intensities, followed by the proposed flare and haze removal components that leverage long- and short-term feature learning to handle the respective degradations. Such a design offers an targeted and effective solution to eliminating various types of degradation in UDC systems. We further extend our design into multi-scale to overcome the scale-changing of degradation that often occur in long-range videos. To demonstrate the superiority of D$^2$RNet, we propose a large-scale UDC video benchmark by gathering HDR videos and generating realistically degraded videos using the point spread function measured by a commercial UDC system. Extensive quantitative and qualitative evaluations demonstrate the superiority of D$^2$RNet compared to other state-of-the-art video restoration and UDC image restoration methods. Code is available at https://github.com/ChengxuLiu/DDRNet.git
BibTeX
@inproceedings{liu2024decoupling,
  author = {Chengxu Liu and Xuan Wang and Yuanting Fan and Shuai Li and Xueming Qian},
  title = {{Decoupling Degradations with Recurrent Network for Video Restoration in Under-Display Camera}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i4.28144},
}
Figure