Fixed Pattern Noise Removal for Multi-View Single-Sensor Infrared Camera

Arnaud Barral, Pablo Arias, Axel DavyWACV 2024

TL;DR

This paper introduces a novel approach to remove Fixed Pattern Noise (FPN) from infrared cameras by leveraging multiple views captured by a single sensor. By generalizing the FPN removal problem to a multi-view setting, they achieve significantly more accurate FPN estimation and removal compared to traditional single-view methods, even enabling state-of-the-art performance for single-view FPN removal through a rolling window technique.

What Problem Does This Paper Address?

Infrared cameras, particularly those using microbolometers, are susceptible to Fixed Pattern Noise (FPN). This noise arises from slight variations in the sensor's response and is temporally coherent, meaning it remains relatively constant over time. FPN significantly degrades image quality and hinders subsequent image processing tasks like tracking and motion estimation. While existing methods attempt to remove FPN, they often struggle with accuracy, especially in scenes with little motion, or require frequent recalibration as the FPN can slowly drift over time. This paper addresses the challenge of more robust and accurate FPN removal by exploiting a specific camera setup: a single sensor capturing multiple views of the same scene, often achieved through periodic motion of a mirror or the camera itself. This multi-view scenario provides richer information to disentangle the FPN from the actual scene content.

Methods

The core of the paper is a novel energy minimization framework for FPN removal. The proposed energy function incorporates data attachment terms (ensuring the denoised image is close to the noisy observation) and spatial regularization (Total Variation, TV) to promote piecewise smooth solutions.

The key innovation lies in generalizing this energy to a multi-view setting. Instead of a single noisy image, the method considers N synchronized sequences, all affected by the same FPN. This allows for a more accurate estimation of the FPN by exploiting the redundancies across these views.

The authors propose two optimization algorithms to minimize this energy:

  • Adam Optimizer: A widely used adaptive gradient descent method, applied here with automatic differentiation.
  • Chambolle-Pock Algorithm: A first-order primal-dual optimization algorithm, particularly well-suited for convex energies with non-differentiable regularization terms like TV.

Furthermore, they adapt their framework for different scenarios:

  • Multi-view Offline FPN Removal: For static FPN estimation, using multiple distinct sequences or frames.
  • Multi-view Online FPN Removal: For slowly varying FPN, by performing online optimization on synchronized sequences.
  • Single-view Online FPN Removal: By employing a rolling window approach, where the method considers the N most recent frames of a single video to estimate the FPN at the current time step. This adaptation allows their method to achieve state-of-the-art results in the standard single-view FPN removal problem.

Datasets

The paper evaluates its methods on both synthetic and real-world datasets:

  • Synthetic Data: Images from the DIV2K dataset were used, converted to grayscale and corrupted with simulated additive FPN (both structured and unstructured components).
  • Real Data: Sequences from the REDS dataset were used for synthetic experiments, and real infrared videos from a microbolometer camera with a cyclic motion pattern were used for multi-view and online evaluations.

No new benchmarks were introduced; instead, the authors compare their method against existing literature on established datasets.

Key Insights and Ideas

  • Multi-view synergy for FPN: The core insight is that having multiple views of the same scene, even with the same FPN, provides significantly more information to accurately estimate and remove the noise compared to relying on a single view. This is because the FPN is constant across views, while the scene content varies, allowing for better disentanglement.
  • Generalization to single-view: The proposed multi-view framework can be effectively adapted to the standard single-view FPN removal problem using a rolling window. This demonstrates the robustness and flexibility of their energy formulation.
  • Convexity and optimization: The proposed energy function is convex, guaranteeing a unique minimizer. The comparison between Adam and Chambolle-Pock highlights the effectiveness of specialized algorithms for such problems, with Chambolle-Pock showing comparable or better results due to its suitability for non-differentiable regularization.
  • Parameter robustness: The method requires minimal parameter tuning, making it adaptable to different camera sensors and noise characteristics, a significant advantage over data-hungry deep learning methods.

Key Results

  • Multi-view Offline: The method achieves significant PSNR improvements on synthetic data, with performance increasing with the number of views (N). Both Adam and Chambolle-Pock yield comparable, high-quality results.
  • Multi-view Online: The proposed method outperforms baseline methods (SL-THPF, bilateral THPF) in multi-view online FPN estimation on both synthetic and real data, producing smoother results with less residual FPN.
  • Single-view Online: The rolling window approach achieves state-of-the-art PSNR results on REDS dataset sequences, outperforming existing methods like SL-THPF, bilateral THPF, and even a single-frame deep learning network (SNRWDNN) after a short convergence period. The method also demonstrates better tracking of slowly varying FPN.
  • Robustness to temporal noise: The FPN removal method can be effectively used as a pre-processing step before AWGN denoising, leading to significantly improved overall denoising performance.

Limitations and Future Work

  • Synchronization Assumption: The multi-view methods assume synchronized sequences. While this is often achievable with controlled camera setups, it might be a limitation for less controlled scenarios.
  • Computational Cost: While the rolling window approach significantly reduces computational cost for single-view FPN removal, the multi-view offline and online methods' computational cost scales linearly with the number of views (N).
  • Parameter Tuning: While the paper states minimal parameter tuning is required, the optimal choice of the regularization parameter ($\lambda$) and step sizes for the optimizers still needs to be determined, though they provide guidance based on noise levels and convergence.

Future work could explore more advanced synchronization techniques or investigate methods that are less sensitive to precise synchronization. Further research into adaptive parameter selection could also enhance the method's autonomy.

Summary updated

BibTeX
@inproceedings{barral2024fixed,
  author = {Arnaud Barral and Pablo Arias and Axel Davy},
  title = {{Fixed Pattern Noise Removal for Multi-View Single-Sensor Infrared Camera}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure