Polarized Color Image Denoising

Zhuoxiao Li, Haiyang Jiang, Mingdeng Cao, Yinqiang ZhengCVPR 2023 · OpenReview

TL;DR

This paper introduces "Pocoformer," a novel Transformer-based deep learning model for denoising polarized color images. By leveraging a new real-world dataset and a hybrid attention mechanism, Pocoformer effectively restores both image clarity and crucial polarization information, outperforming existing methods.

What Problem Does This Paper Address?

Polarized color photography offers rich information about object materials and surface geometry, making it valuable for applications like reflection removal and 3D reconstruction. However, a key challenge arises from the design of single-chip polarized color sensors. These sensors use a polarizing filter array in front of the image sensor, which significantly reduces the amount of light captured, leading to very low Signal-to-Noise Ratio (SNR), especially in low-light conditions. This noise degrades the visual quality of the images and, more importantly, corrupts the polarization information, hindering downstream analysis. Traditional image denoising methods struggle with this problem due to the complex physical constraints inherent in polarized imaging.

Methods

The core of this work is the Pocoformer model, a U-shaped Transformer architecture designed for polarized color image denoising. It utilizes Hybrid Transformer Blocks (HTBs), which combine two novel attention mechanisms:

  • Shifted Window Multi-head Attention (SW-MA): This mechanism captures spatial dependencies efficiently by operating within local windows that shift across layers, offering a better speed-accuracy trade-off than global attention for high-resolution images.
  • Window-based Multi-Shuffled-heads Transposed Attention (W-MSTA): This attention mechanism focuses on inter-channel interactions, crucial for preserving polarization information. It employs a channel shuffle operation to enhance connections between channel groups.

The model also incorporates a Smoothing MLP to mitigate border artifacts introduced by window-based attention. The overall architecture consists of encoder and decoder stages with downsampling and upsampling layers, allowing it to handle multi-scale features. The loss function is a combination of a pixel-wise L1 loss and a Stoke parameter loss (Ls), which specifically constrains the preservation of polarization information.

Datasets

The authors introduce a real-world polarized color image dataset for training and evaluation. This dataset consists of 106 scenes, with each scene providing a pair of:

  • Noisy, short-exposure raw images: Captured under low-light conditions.
  • Clean, long-exposure reference images: Used as ground truth.

The dataset is captured using a FLIR BFS-U3-51S5PC polarized color camera. Additionally, an evaluation set from a different camera, the LUCID VISION TRI050S1-QC with an IMX264MYR sensor, is used to test the model's generalization capabilities.

Key Insights and Ideas

  • The critical need for preserving polarization information: Simply denoising the image intensity (So) is insufficient. The paper highlights that traditional methods often destroy or corrupt the Degree of Linear Polarization (DoLP, ρ) and Angle of Polarization (AoP, φ), which are essential for many applications.
  • Transformer's suitability for polarized image tasks: The paper demonstrates that Transformer architectures, with their ability to capture global context and long-range dependencies, are well-suited for the complex interactions within polarized color images.
  • Hybrid attention for dual objectives: The combination of spatial (SW-MA) and channel (W-MSTA) attention mechanisms within the HTB is key to simultaneously achieving noise reduction and polarization information restoration.
  • The impact of ISP processing: The paper emphasizes that non-linear image signal processing (ISP) steps like gamma correction and tone mapping, as well as quantization, can severely degrade polarization information, underscoring the need for raw data processing.

Key Results

  • State-of-the-art performance: Pocoformer achieves the best results across multiple metrics (PSNR, SSIM for image, ρ, and φ) compared to traditional and recent deep learning methods on their custom dataset.
  • Significant improvement in polarization metrics: The model shows substantial gains in restoring ρ and φ, which are often compromised by other methods. For instance, it achieves a PSNR of 25.32 dB for ρ and 12.34 dB for φ.
  • Computational efficiency: Despite its Transformer backbone, Pocoformer offers a favorable trade-off between performance and computational cost, significantly reducing FLOPs compared to some Transformer-based competitors like SwinIR.
  • Effective generalization: The model demonstrates strong performance on a dataset captured with a different polarized color sensor, indicating good generalization capabilities.
  • Benefit to downstream tasks: Visualizations show that denoised images from Pocoformer lead to improved results in downstream applications like 3D human shape reconstruction and reflection removal.

Limitations and Future Work

  • Computational cost of Transformers: While Pocoformer is more efficient than some Transformer models, it is still computationally more intensive than traditional methods. The paper mentions a lightweight architecture (Ours-S) to address this.
  • Generalization to diverse sensors: While tested on a second sensor, further evaluation across a wider range of polarized color sensors and under more varied environmental conditions would be beneficial.
  • Future research directions: The authors suggest that their work can inspire future research in polarized color image restoration and its applications. They also highlight the potential of their denoising pipeline as a pre-processing step for various downstream tasks.

Summary updated

Abstract

Single-chip polarized color photography provides both visual textures and object surface information in one snapshot. However, the use of an additional directional polarizing filter array tends to lower photon count and SNR, when compared to conventional color imaging. As a result, such a bilayer structure usually leads to unpleasant noisy images and undermines performance of polarization analysis, especially in low-light conditions. It is a challenge for traditional image processing pipelines owing to the fact that the physical constraints exerted implicitly in the channels are excessively complicated. In this paper, we propose to tackle this issue through a noise modeling method for realistic data synthesis and a powerful network structure inspired by vision Transformer. A real-world polarized color image dataset of paired raw short-exposed noisy images and long-exposed reference images is captured for experimental evaluation, which has demonstrated the effectiveness of our approaches for data synthesis and polarized color image denoising. The code and data can be found at https://github.com/bandasyou/pcdenoise.
BibTeX
@inproceedings{li2023polarized,
  author = {Zhuoxiao Li and Haiyang Jiang and Mingdeng Cao and Yinqiang Zheng},
  title = {{Polarized Color Image Denoising}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  url = {https://openreview.net/forum?id=U6rQAzIu0j},
}
Figure