Efficient Polarization Demosaicking via Low-cost Edge-aware and Inter-channel Correlation

Guangsen Liu, Peng Rao, Xin Chen, Yao Li, Haixin JiangarXiv 2024 · DOI · arXiv

TL;DR

Researchers have developed a new three-stage framework for polarization demosaicking that achieves state-of-the-art accuracy while remaining fast enough for real-time industrial use. By replacing rigid "binary" edge detection with a smooth logistic-function-based weighting system and adding an inter-channel refinement step, their "LEIC" algorithm outperforms existing methods in both visual quality and computational efficiency.

What Problem Does This Paper Address?

Division of Focal Plane (DoFP) polarization sensors are the workhorses of modern polarimetric imaging. Unlike older systems that required rotating filters, DoFP sensors use a Polarization Filter Array (PFA)—similar to a color Bayer filter—to capture four different polarization orientations (0°, 45°, 90°, and 135°) simultaneously.

The catch? Each pixel only records one orientation. To get a full polarization profile for every pixel, we must "demosaic" the image—interpolating the missing data. Existing solutions usually fall into two camps: simple interpolation (fast but prone to "zipper" artifacts and blurring) or deep learning/optimization (accurate but too slow for real-time industrial applications). This paper targets the "sweet spot": an algorithm that is mathematically lightweight but sophisticated enough to handle complex edges without losing detail.

Methods

The authors introduce the Three-stage Interpolation-based Polarization Demosaicking Framework (TIPDF), which standardizes the reconstruction process into three logical steps:

  1. Estimating Orthogonal Channel Planes: The algorithm first fills in the missing values for the 0°/90° and 45°/135° pairs.
  2. Determining Non-orthogonal Planes: It then uses the results from the first stage to estimate the remaining channels.
  3. Refining Results: A final calibration step ensures consistency across all channels.

The core innovations within this framework are:

  • DLE (DoFP Low-cost Edge-aware): Traditional edge-aware filters use a "step function" (is there an edge? Yes or No). This often leads to errors. The authors use a Logistic Function to create a continuous, smooth weighting system. This allows the algorithm to make "flexible" decisions about how much to trust diagonal vs. anti-diagonal neighbors.
  • ICCC (Inter-channel Correlation Calibration): Since polarization channels are highly correlated, the authors use a "difference plane" technique (common in color imaging) to refine the initial estimates and eliminate redundant textures.

The authors provide two versions: LEIC (the full version with calibration) and LEPD (a lightweight version that skips the final calibration for maximum speed).

Datasets

The study evaluates the algorithms using two primary sources:

  • Qiu et al. Dataset: A standard dataset of 40 scenes used to synthesize simulated DoFP images for quantitative benchmarking (PSNR, RMSE, SSIM).
  • OLVD (Outdoor Long-wave infrared and Visible DoFP) Dataset: A newly created dataset by the authors. It includes thousands of images captured with real-world LWIR and visible-light polarization cameras, featuring complex outdoor scenes like ships and UAVs.

Key Insights and Ideas

  • The "Smoothness" Advantage: The move from a ternary (0, 0.5, 1) edge classifier to a continuous logistic function is a major "aha" moment. It prevents the "binary" mistakes that lead to the dreaded "zip" effect in high-frequency areas.
  • Hardware-Friendly Design: Both LEIC and LEPD operate within a tight 5×5 pixel window. This locality is crucial; it means the algorithm is perfectly suited for parallel acceleration on GPUs or FPGAs, making it viable for high-speed industrial cameras.
  • Orthogonality Matters: By recognizing that 0° and 90° are orthogonal (and thus carry specific mathematical relationships), the algorithm prioritizes reconstructing these pairs first to stabilize the rest of the interpolation.

Key Results

  • Accuracy: In quantitative tests on simulated images, LEIC achieved the highest PSNR (Signal-to-Noise ratio) and SSIM (Structural Similarity) across almost all polarization metrics (I0, I45, I90, I135, S0, DoLP, and AoLP).
  • Speed: The lightweight LEPD is the fastest non-traditional method tested. It can process a 1-megapixel (1024×1024) image in just 0.1402 seconds on a standard consumer CPU (AMD Ryzen 5600X). The full LEIC version takes 0.2693 seconds, which is still significantly faster than advanced competitors like EARI (1.31s) or NP (0.51s).
  • Visual Quality: Qualitative results on the OLVD dataset show that LEIC effectively suppresses "x-like" artifacts and false textures that plague other edge-aware methods like Newton’s Polynomial (NP) interpolation.

Limitations and Future Work

While the 5×5 window ensures speed, it may limit the algorithm's ability to reconstruct very large, complex structures that span wider areas of the image compared to deep learning models with larger receptive fields. Additionally, while the OLVD dataset is a great step forward, the authors note that acquiring perfectly aligned "ground truth" images for real-world DoFP sensors remains a significant challenge for the entire field. Future work likely involves further optimizing the ICCC step to reduce its computational overhead even further.

Summary updated

Abstract

Efficient and high-fidelity polarization demosaicking is critical for industrial applications of the division of focal plane (DoFP) polarization imaging systems. However, existing methods have an unsatisfactory balance of speed, accuracy, and complexity. This study introduces a novel polarization demosaicking algorithm that interpolates within a three-stage basic demosaicking framework to obtain DoFP images. Our method incorporates a DoFP low-cost edge-aware technique (DLE) to guide the interpolation process. Furthermore, the inter-channel correlation is used to calibrate the initial estimate in the polarization difference domain. The proposed algorithm is available in both a lightweight and a full version, tailored to different application requirements. Experiments on simulated and real DoFP images demonstrate that our two methods have the highest interpolation accuracy and speed, respectively, and significantly enhance the visuals. Both versions efficiently process a 1024*1024 image on an AMD Ryzen 5600X CPU in 0.1402s and 0.2693s, respectively. Additionally, since our methods only involve computational processes within a 5*5 window, the potential for parallel acceleration on GPUs or FPGAs is highly feasible.
BibTeX
@misc{liu2024efficient,
  author = {Guangsen Liu and Peng Rao and Xin Chen and Yao Li and Haixin Jiang},
  title = {{Efficient Polarization Demosaicking via Low-cost Edge-aware and Inter-channel Correlation}},
  year = {2024},
  eprint = {2408.17099},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.17099},
}
Figure