Zero-Reference Image Restoration for Under-Display Camera of UAV

Zhuoran Zheng, Xiuyi Jia, Yunliang ZhuangarXiv 2022 · arXiv

Abstract

Unmanned Aerial Vehicles (UAVs) operating in challenging environments often utilize Transparent OLED (T-OLED) screens to protect camera hardware. However, these screens introduce significant image degradation, characterized by a "white veil" effect stemming from diffraction and atmospheric haze. Traditional restoration techniques rely on paired "clear" and "degraded" images, which are nearly impossible to capture synchronously in unstable UAV flight conditions. This paper introduces ZRUDC-Net, a lightweight, zero-reference network designed for real-time image restoration. By leveraging a localized affine grid estimation and a suite of non-reference loss functions, ZRUDC-Net restores high-quality visuals without requiring ground-truth data. Experimental results demonstrate state-of-the-art performance on T-OLED benchmarks and real-time processing speeds of 18ms for 2K resolution images.

Introduction

The integration of cameras within UAVs presents a trade-off between hardware protection and image quality. To shield sensitive lenses from wind, debris, and mechanical stress, engineers have proposed "all-in-one" designs where the camera is placed behind a T-OLED display. While aesthetically pleasing and protective, this configuration introduces a persistent "white veil" effect. This degradation is caused by a combination of low light transmission through the OLED substrate, diffraction patterns, and atmospheric haze exacerbated by temperature gradients between the ground and sky.

UAV camera configurations and the resulting 'white veil' effect.

Existing restoration methods typically utilize supervised learning, requiring large datasets of perfectly aligned paired images. For UAVs, capturing such data is impractical due to flight instability and changing environmental conditions. Consequently, there is a critical need for unsupervised, zero-reference restoration models that can operate efficiently on-board.

Methodology: ZRUDC-Net

The proposed ZRUDC-Net (Zero-Reference Under-Display Camera Network) addresses restoration as an image-specific affine estimation problem. The architecture is optimized for speed, ensuring it can handle live video feeds from high-resolution UAV sensors.

Network Architecture

The pipeline consists of four distinct stages:

  1. Downsampling: To maintain computational efficiency, the input image is first resized to a fixed $256 \times 256$ resolution using bilinear downsampling.
  2. Affine Grid Estimation: A modified U-Net processes the downsampled image to generate a low-rank affine grid (a $12 \times 256 \times 256$ tensor). This grid serves as a localized transformation map, defining how color and texture should be adjusted across the image.
  3. Regularization: To prevent artifacts and ensure spatial smoothness, a 2D MaxPool layer is applied to the estimated grid.
  4. Smoothing Slicing (S-Slicing): The regularized grid is upsampled back to the original input resolution. The final restoration is performed by applying the grid to the raw input image via a learned convolution operator, producing the "clear" output.
The ZRUDC-Net architecture and S-Slicing pseudo-code.

Zero-Reference Training

Unlike standard networks, ZRUDC-Net is trained using a "basket" of non-reference loss functions. These include:

  • Spatial Consistency Loss: Maintains the structural integrity of the image.
  • Exposure Control Loss: Corrects for the overexposure often caused by the T-OLED screen.
  • Color Constancy Loss: Ensures natural color reproduction.
  • Dark Channel Prior (DCP) Loss: Specifically targets and removes the haze-like "white veil."

Datasets and Evaluation

The model was evaluated using two datasets:

  • UAV-UDC Dataset: A novel collection of 30,000 frames from 15 UAV video clips, covering diverse daytime and nighttime scenes.
  • T-OLED Dataset: A public benchmark (Zhou et al., 2021) used for comparative analysis against state-of-the-art (SOTA) methods.

Results and Discussion

ZRUDC-Net achieves a balance between restoration quality and inference speed that surpasses existing models.

Quantitative Performance

On the T-OLED benchmark, ZRUDC-Net achieved a Peak Signal-to-Noise Ratio (PSNR) of 37.77 and a Structural Similarity Index (SSIM) of 0.9689. These metrics outperform established methods like HDRNet and DEUNet, which often struggle with the specific artifacts introduced by under-display configurations.

Real-Time Efficiency

A key advantage of ZRUDC-Net is its efficiency. By estimating a low-rank grid rather than performing pixel-wise mapping, the model processes 2K resolution images in approximately 18ms. This performance makes it the fastest model in its class, suitable for integration into real-time UAV control systems.

Qualitative Analysis

Visual inspections confirm that the model effectively eliminates the "white veil" and restores vivid colors. Unlike other unsupervised methods, ZRUDC-Net avoids common pitfalls such as over-enhancement or the creation of "dark corner" artifacts, maintaining a natural appearance across the frame.

Conclusion and Future Work

ZRUDC-Net provides a robust solution for the unique challenges of under-display UAV cameras. By bypassing the need for paired training data and focusing on efficient affine grid estimation, it enables high-quality, real-time restoration in unpredictable environments.

However, limitations remain. The model's performance degrades in extreme low-light conditions where the sensor fails to capture sufficient illumination data. Additionally, while optimized for T-OLED, different display types like P-LED may require further architectural adaptations to handle varying diffraction patterns. Future research will explore more sophisticated pooling mechanisms to refine the affine grid without compromising the model's real-time capabilities.

Summary updated

Abstract

The exposed cameras of UAV can shake, shift, or even malfunction under the influence of harsh weather, while the add-on devices (Dupont lines) are very vulnerable to damage. We can place a low-cost T-OLED overlay around the camera to protect it, but this would also introduce image degradation issues. In particular, the temperature variations in the atmosphere can create mist that adsorbs to the T-OLED, which can cause secondary disasters (i.e., more severe image degradation) during the UAV's filming process. To solve the image degradation problem caused by overlaying T-OLEDs, in this paper we propose a new method to enhance the visual experience by enhancing the texture and color of images. Specifically, our method trains a lightweight network to estimate a low-rank affine grid on the input image, and then utilizes the grid to enhance the input image at block granularity. The advantages of our method are that no reference image is required and the loss function is developed from visual experience. In addition, our model can perform high-quality recovery of images of arbitrary resolution in real time. In the end, the limitations of our model and the collected datasets (including the daytime and nighttime scenes) are discussed.
BibTeX
@misc{zheng2022zero,
  author = {Zhuoran Zheng and Xiuyi Jia and Yunliang Zhuang},
  title = {{Zero-Reference Image Restoration for Under-Display Camera of UAV}},
  year = {2022},
  eprint = {2202.06283},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2202.06283},
}
Figure