Decoupling Degradations with Recurrent Network for Video Restoration in Under-Display Camera

Chengxu Liu, Xuan Wang, Yuanting Fan, Shuai Li, Xueming Qian

Introduction

Under-Display Cameras (UDC) represent a significant milestone for full-screen smartphone design, allowing the selfie camera to be hidden beneath the OLED display. However, this placement introduces severe image degradation. The OLED pixel grid acts as a diffraction grating, causing light to scatter and attenuate before reaching the sensor. This results in two primary artifacts: flare (intense glare in bright regions leading to content loss) and haze (low-contrast blur in darker regions).

While single-image restoration has improved, video restoration remains difficult because the Point Spread Function (PSF) changes dynamically with motion. Standard video restoration models often fail because they treat all degradations uniformly, ignoring the distinct temporal requirements for fixing flare versus haze.

The D2RNet Framework

To address these challenges, the authors introduce D2RNet, the first specialized video restoration framework for UDC. The core philosophy of D2RNet is "decoupling": separating the complex UDC degradation into its constituent flare and haze components to apply targeted restoration strategies.

D2RNet framework overview showing the decoupling of flare and haze maps for targeted restoration.

As shown in the framework overview, the system generates a Flare Map and a Haze Map from the input features. This allows the model to handle flare through Long-Term Flare Removal, which searches across distant frames to recover lost content, and Short-Term Haze Removal, which utilizes adjacent frames for local spatio-temporal sharpening.

Architecture and Decoupling Attention

D2RNet is built as a multi-scale, bi-directional recurrent network. The engine of this network is the Decoupling Attention Module (DAM).

Detailed architecture of D2RNet featuring the Decoupling Attention Module (DAM) and bi-directional recurrent propagation.

The DAM uses a soft mask generation function based on pixel intensity to split features. By processing the video in both forward and backward directions across three scales ($2\times, 4\times, 8\times$ down-sampling), the network effectively gathers information from the entire sequence. This architecture ensures that the model can handle objects of varying sizes and recover details even when they are temporarily obscured by intense light sources.

Data and Performance

A significant hurdle in UDC research is the lack of large-scale video datasets. The authors addressed this by creating VidUDC33K, a dataset of 677 high-quality HDR videos. They simulated UDC degradation by measuring the real-world PSF of a ZTE Axon 20 system and applying it to the HDR frames using a physics-based model that accounts for motion and saturation.

In benchmarks, D2RNet achieved a state-of-the-art 31.91 dB PSNR on VidUDC33K, outperforming general video restoration models like RVRT by over 1 dB. Notably, D2RNet is highly efficient, utilizing only 5.76M parameters and processing frames in 0.44s, making it significantly faster and lighter than transformer-based alternatives like VRT.

Conclusion

D2RNet demonstrates that non-uniform degradations in UDC video are best handled through specialized decoupling. By recognizing that flare and haze require different temporal "memories," the model achieves superior restoration quality with high efficiency. This approach sets a new benchmark for UDC video processing and offers a blueprint for handling complex, intensity-dependent degradations in other imaging domains.

Summary updated

Abstract

Under-display camera (UDC) systems are the foundation of full-screen display devices in which the lens mounts under the display. The pixel array of light-emitting diodes used for display diffracts and attenuates incident light, causing various degradations as the light intensity changes. Unlike general video restoration which recovers video by treating different degradation factors equally, video restoration for UDC systems is more challenging that concerns removing diverse degradation over time while preserving temporal consistency. In this paper, we introduce a novel video restoration network, called D$^2$RNet, specifically designed for UDC systems. It employs a set of Decoupling Attention Modules (DAM) that effectively separate the various video degradation factors. More specifically, a soft mask generation function is proposed to formulate each frame into flare and haze based on the diffraction arising from incident light of different intensities, followed by the proposed flare and haze removal components that leverage long- and short-term feature learning to handle the respective degradations. Such a design offers an targeted and effective solution to eliminating various types of degradation in UDC systems. We further extend our design into multi-scale to overcome the scale-changing of degradation that often occur in long-range videos. To demonstrate the superiority of D$^2$RNet, we propose a large-scale UDC video benchmark by gathering HDR videos and generating realistically degraded videos using the point spread function measured by a commercial UDC system. Extensive quantitative and qualitative evaluations demonstrate the superiority of D$^2$RNet compared to other state-of-the-art video restoration and UDC image restoration methods. Code is available at https://github.com/ChengxuLiu/DDRNet.git
BibTeX
@inproceedings{liu2024decoupling,
  author = {Chengxu Liu and Xuan Wang and Yuanting Fan and Shuai Li and Xueming Qian},
  title = {{Decoupling Degradations with Recurrent Network for Video Restoration in Under-Display Camera}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i4.28144},
}
Figure