Joint Demosaicking / Rectification of Fisheye Camera Images using Multi-color Graph Laplacian Regularization

Fengbo Lan, Cheng Yang, Gene Cheung, Jack Z. G. Tan

Abstract

To compose a 360 image from a rig with multiple fisheye cameras, a conventional processing pipeline first performs demosaicking on each fisheye camera's Bayer-patterned grid, then translates demosaicked pixels from the camera grid to a rectified image grid—thus performing two image interpolation steps in sequence. Hence interpolation errors can accumulate, and acquisition noise in the captured pixels can pollute neighbors in two consecutive processing stages. In this paper, we propose a joint processing framework that performs demosaicking and grid-to-grid mapping simultaneously—thus limiting noise pollution to one interpolation. Specifically, we first obtain a reverse mapping function from a regular on-grid location in the rectified image to an irregular off-grid location in the camera's Bayer-patterned image. For each pair of adjacent pixels in the rectified grid, we estimate its gradient using the pair's neighboring pixel gradients in three colors in the Bayer-patterned grid. We construct a similarity graph based on the estimated gradients, and interpolate pixels in the rectified grid directly via graph Laplacian regularization (GLR). Experiments show that our joint method outperforms several competing local methods that execute demosaicking and rectification in sequence, by up to 0.52 dB in PSNR and 0.086 in SSIM on the publicly available dataset, and by up to 5.53dB in PSNR and 0.411 in SSIM on the in-house constructed dataset.

Introduction

In the production of 360° virtual reality content, fisheye camera rigs are essential for capturing wide-angle perspectives. However, the raw data from these sensors requires significant processing before it can be viewed. Traditionally, this involves a two-step sequential pipeline: first, demosaicking to interpolate missing color values from the Bayer-patterned sensor, and second, rectification to map those pixels from a distorted fisheye grid to a flat, rectilinear grid.

The primary drawback of this sequential approach is the accumulation of interpolation errors. Each stage introduces its own artifacts, and sensor noise present in the raw data is effectively "smeared" across neighboring pixels twice. This results in a loss of high-frequency detail and amplified noise in the final output. This paper proposes a joint framework that executes both tasks in a single step, ensuring that interpolation occurs only once, thereby preserving image fidelity.

Methodology

The proposed framework leverages Graph Signal Processing (GSP) to handle the irregular mapping between the raw sensor and the target rectified grid.

Reverse Mapping and Gradient Estimation

Unlike forward mapping, which can leave holes in the target image, the algorithm utilizes a reverse mapping function. For every regular grid position in the final rectified image, the system identifies the corresponding (often non-integer) location on the raw Bayer sensor.

Illustration of the reverse mapping process from the rectified image grid back to the irregular locations on the Bayer-patterned sensor grid.

To preserve sharp edges, the method estimates local gradients using a Maximum Likelihood Estimation (MLE) approach. This estimation accounts for the strong correlations between Red, Green, and Blue color channels in natural scenes, allowing the model to detect edges more accurately than single-color methods.

Graph Laplacian Regularization (GLR)

The image is modeled as a signal on a graph where edges represent pixel similarities. The reconstruction is formulated as an optimization problem using Graph Laplacian Regularization. The goal is to find a rectified image that minimizes a cost function balancing data fidelity (consistency with the raw sensor) and smoothness (adherence to the estimated graph structure). This is solved iteratively using the conjugate gradient method, which is well-suited for large-scale quadratic programming problems.

Experimental Results

The performance of the joint method was evaluated against standard sequential pipelines, specifically Bilinear interpolation and High-Quality Linear (HQL) interpolation, across two datasets: the publicly available Multi-FoV dataset and a custom in-house dataset featuring complex 3D models.

Quantitative performance comparison (SSIM and PSNR) between the proposed joint method and standard sequential baselines.

Quantitative Analysis

As shown in the results table, the proposed method consistently outperforms the baselines. On the Multi-FoV dataset (scenes like "room" and "city"), the method achieved gains of up to 0.52 dB in PSNR. The improvements were even more pronounced on the in-house dataset, where the joint approach outperformed sequential methods by up to 5.53 dB in PSNR and 0.411 in SSIM. These metrics indicate significantly higher structural similarity to the ground truth and lower overall noise.

Qualitative Observations

Visual inspections confirm the quantitative data. The joint method effectively eliminates "zipper" artifacts commonly found at high-contrast edges in demosaicked images. Furthermore, it maintains sharper boundaries and produces cleaner flat regions compared to sequential pipelines, which often exhibit visible blur and noise amplification.

Conclusion and Future Work

By unifying demosaicking and rectification into a single optimization problem, this research demonstrates that "one interpolation is better than two." The use of Graph Laplacian Regularization allows for a mathematically robust way to handle the irregular geometries inherent in fisheye lens correction.

Future work will focus on reducing the computational complexity of the iterative solver to enable real-time processing on mobile VR hardware. Additionally, the authors plan to explore non-linear interpolation methods for the initial mapping matrix to further enhance reconstruction accuracy in highly distorted regions of the fisheye frame.

Summary updated

Abstract

To compose a 360 image from a rig with multiple fisheye cameras, a conventional processing pipeline first performs demosaicking on each fisheye camera's Bayer-patterned grid, then translates demosaicked pixels from the camera grid to a rectified image grid---thus performing two image interpolation steps in sequence. Hence interpolation errors can accumulate, and acquisition noise in the captured pixels can pollute neighbors in two consecutive processing stages. In this paper, we propose a joint processing framework that performs demosaicking and grid-to-grid mapping simultaneously---thus limiting noise pollution to one interpolation. Specifically, we first obtain a reverse mapping function from a regular on-grid location in the rectified image to an irregular off-grid location in the camera's Bayer-patterned image. For each pair of adjacent pixels in the rectified grid, we estimate its gradient using the pair's neighboring pixel gradients in three colors in the Bayer-patterned grid. We construct a similarity graph based on the estimated gradients, and interpolate pixels in the rectified grid directly via graph Laplacian regularization (GLR). Experiments show that our joint method outperforms several competing local methods that execute demosaicking and rectification in sequence, by up to 0.52 dB in PSNR and 0.086 in SSIM on the publicly available dataset, and by up to 5.53dB in PSNR and 0.411 in SSIM on the in-house constructed dataset.
BibTeX
@inproceedings{lan2020joint,
  author = {Fengbo Lan and Cheng Yang and Gene Cheung and Jack Z. G. Tan},
  title = {{Joint Demosaicking / Rectification of Fisheye Camera Images using Multi-color Graph Laplacian Regularization}},
  booktitle = {ICIP},
  year = {2020},
  url = {https://openreview.net/forum?id=SZPJvIcejF},
}
Figure