TL;DR
This paper introduces a novel adaptive deghosting method for infrared detectors that uses a dynamic learning rate rule. By combining temporal gating with an adaptive spatial threshold based on edge detection and noise estimation, the method effectively suppresses ghosting artifacts without sacrificing convergence speed, outperforming existing neural network-based nonuniformity correction techniques.
What Problem Does This Paper Address?
Infrared focal plane arrays (IRFPAs) are crucial for various applications, but their performance is degraded by fixed-pattern noise (FPN), also known as nonuniformity. While scene-based nonuniformity correction (SBNUC) algorithms, particularly those based on neural networks (NNs), are promising, they face two major challenges: slow convergence and the generation of "ghosting artifacts." Ghosting occurs when previous scene information incorrectly influences the correction of the current frame, leading to persistent, ghost-like remnants. Existing methods often struggle to balance the suppression of ghosting with maintaining a fast convergence rate, and their fixed thresholds can be ineffective in varying noise conditions. This paper aims to address this trade-off by developing an adaptive approach that dynamically adjusts its parameters to the image content and noise levels.
Methods
The core of the proposed method is an adaptive learning rate rule that integrates two key components:
- Combined Temporal Gate and Edge Detection: This module controls when the correction parameters are updated. It uses a temporal gate to detect motion and an edge detection mechanism (specifically, the Sobel operator) to identify scene textures. The learning rate is suppressed in areas with significant scene edges and motion, preventing the propagation of ghosting artifacts.
- Adaptive Spatial Threshold: This is the novel contribution. Instead of using a fixed threshold for edge detection, this method dynamically adjusts it based on the estimated residual nonuniformity noise. This is achieved by:
- Noise Estimation: Analyzing the gradient of the correction matrix and the local variance of the image to estimate the noise level.
- Two-Stage Correction: The algorithm operates in two stages: a "denoising stage" with a larger adaptive threshold to aggressively remove FPN, and a "deghosting stage" with a smaller adaptive threshold to preserve image details and suppress ghosting. A flag, STA, determines which stage is active, based on the temporal variation of the correction matrix gradient.
The overall algorithm uses a Least Mean Square (LMS) approach for updating the correction parameters, but the learning rate (step size, denoted by $\epsilon$) is dynamically controlled by the proposed adaptive mechanism.
Datasets
The paper evaluates the proposed method using two types of datasets:
- Simulated Nonuniformity Data: A clean infrared video sequence from a 336 × 256 FPA was corrupted with simulated global synthetic offset and vertical stripe offset. The sequence contained 1000 frames and was designed to induce ghosting artifacts through motion and sudden scene changes.
- Real Nonuniformity Data:
- A 320 × 240 InSb FPA dataset was used, featuring an indoor scene with severe FPN. This sequence had 2000 frames and involved continuous motion, followed by hand occlusion of the field of view.
- A 384 × 288 FPA dataset from a different imager was used, with a noisier outdoor scene and random imager motion. This sequence comprised 3315 frames.
Key Insights and Ideas
- Adaptive Threshold is Key: The most significant insight is that a fixed spatial threshold is insufficient for robust deghosting across varying noise and scene conditions. An adaptive threshold, informed by the actual residual noise, is crucial for balancing denoising and deghosting.
- Two-Stage Approach for Different Goals: Separating the correction process into distinct denoising and deghosting stages, each with its own adaptive threshold strategy, allows the algorithm to prioritize different objectives at different points in the correction process.
- Gradient of Correction Matrix as Convergence Indicator: The temporal variation of the correction matrix gradient serves as an effective indicator of convergence and the severity of residual noise, enabling the switch between denoising and deghosting stages.
- Sobel Operator's Efficiency: The choice of the Sobel operator for edge detection is highlighted as a good balance between preserving primitive gradient information and maintaining computational efficiency compared to more complex methods like Canny.
Key Results
- Superior Deghosting: The proposed method demonstrated significantly fewer ghosting artifacts compared to existing Gated-LMS and ST-LMS algorithms, both in simulated and real-world infrared sequences. This was visually confirmed and quantitatively supported by PSNR (for simulated data) and roughness index (for real data).
- Equivalent Convergence Speed: The adaptive nature of the proposed method did not hinder the convergence speed. In simulated data, its PSNR performance was comparable to the other methods during stable motion and superior during jittering.
- Robustness to Noise: The algorithm performed well even with severe FPN and noisy real-world data, maintaining good deghosting ability without compromising NUC performance.
- Quantitative Improvements:
- In simulated data, the proposed method achieved higher PSNR, especially during imager jittering, where other methods showed significant downward spikes due to ghosting.
- In real data experiments, the roughness index (lower is better) for the proposed method was consistently lower than or comparable to the other methods, indicating better FPN removal and less degradation.
Limitations and Future Work
- Computational Cost: The paper acknowledges that the adaptive edge detection module increases computational complexity and runtime compared to simpler methods. While the division operation can be optimized, the overall processing time is higher.
- Circuit Design Complexity: Implementing the proposed algorithm in hardware (e.g., FPGAs) would increase power consumption and design difficulty due to the added complexity of edge detection and adaptive threshold calculation.
- Future Work Suggestions:
- Investigate the use of machine learning methods to further improve the accuracy of incident radiation estimation.
- Explore ways to mitigate the increased power consumption and design complexity for embedded systems.
- Further refine the adaptive thresholding mechanism to handle more complex scene dynamics and noise patterns.
Summary updated