A Method of Sky Ripple Residual Nonuniformity Reduction for a Cooled Infrared Imager and Hardware Implementation

Yiyang Li, Weiqi Jin, Shuo Li, Xu Zhang, Jin ZhuSensors 2017 · DOI

TL;DR

This paper introduces an improved nonuniformity correction (NUC) algorithm for cooled infrared imagers that specifically targets "sky ripple" noise. By using fuzzy scene classification to adaptively adjust correction thresholds, the method effectively removes this noise while preserving image details and avoiding ghosting artifacts, outperforming existing techniques. The algorithm has also been successfully implemented on an FPGA for real-time processing.

What Problem Does This Paper Address?

Infrared (IR) imagers, particularly cooled ones, often suffer from a specific type of noise called "ripple residual nonuniformity" (RNU) when observing sky scenes. This noise is characterized by a low-frequency, spatially continuous ripple that significantly degrades image quality, making it difficult to detect small targets. Traditional calibration-based NUC methods struggle with this noise, especially when the observed radiation falls outside the calibration range (common in sky observations). Existing scene-based NUC algorithms also face challenges, either being computationally expensive, prone to ghosting artifacts, or having difficulty adapting to the variable nature of ripple RNU. The core problem is to effectively remove this specific type of noise without sacrificing crucial image details or introducing new artifacts, especially for applications like small target detection.

Methods

The paper proposes an Improved Temporal High-Pass & Grayscale Mapping (ITHP & GM) algorithm. This method builds upon the existing THP & GM algorithm by incorporating a fuzzy scene classification module.

Here's a breakdown of the key components:

  • Adaptive Thresholding: The core innovation is the adaptive control of correction thresholds. Instead of using fixed thresholds for temporal (Tte) and spatial (Tsp) filtering, these thresholds are dynamically adjusted based on the scene content.
  • Fuzzy Scene Classification: A fuzzy logic system is employed to classify the scene into three categories: SKY, HALF-SKY, and GROUND. This classification is based on statistical properties of image blocks, such as mean grayscale and grayscale differences between blocks. The output of this classifier is a "sky similarity" value.
  • Threshold Adjustment: The "sky similarity" value directly influences the temporal and spatial thresholds. Higher sky similarity leads to larger thresholds, enabling more aggressive correction for ripple RNU in sky-like scenes. Conversely, lower sky similarity results in smaller thresholds, preserving details in ground scenes and preventing ghosting.
  • Hardware Implementation: The algorithm has been implemented on a Xilinx Virtex-5 XC5VLX50T FPGA. This implementation focuses on efficient processing with low resource consumption and minimal hardware delay (less than 10 image rows).

Datasets

The paper uses real-world infrared video sequences for evaluation.

  • Scene Classification Dataset: A collection of infrared images from August 2015, including deep-sky, mid-sky, near-ground-sky, ground, and indoor scenes. Each scene type has 5000 frames.
  • RNU Correction Dataset: A sequence of 450 frames from a calibrated infrared imager (320x256 detector, 16-bit, 50 fps) that captured outdoor scenes, with ripple RNU appearing around the 260th frame.
  • Pure Sky Dataset: A sequence of 520 pure sky images with almost uniform radiation, some containing small flying objects, used to evaluate convergence rate.

The paper also mentions using a separate image sequence of 9044 frames collected in June 2016 for further verification of the fuzzy classifier's accuracy.

Key Insights and Ideas

  • The "Sky Ripple" is a Distinct Problem: The paper clearly identifies ripple RNU as a specific challenge for cooled IR imagers observing the sky, distinct from general FPN.
  • Adaptive Thresholding is Crucial: The success of the proposed method hinges on the adaptive control of correction thresholds. Fixed thresholds, as used in prior methods, lead to either insufficient noise removal or significant detail degradation.
  • Fuzzy Logic for Robust Scene Understanding: Fuzzy logic provides a flexible and resilient way to classify scenes, especially for transitional "half-sky" scenarios, which is critical for accurate threshold adaptation. This is more effective than simple binary classification.
  • Trade-off Management: The core idea is to manage the inherent trade-off between noise suppression and detail preservation by dynamically adjusting the correction aggressiveness based on the scene's sky-like characteristics.
  • FPGA Implementation for Real-Time Needs: The successful hardware implementation demonstrates the algorithm's suitability for real-time applications in infrared imaging systems, offering low latency and efficient resource usage.

Key Results

  • Improved Detail Preservation: The ITHP & GM method significantly preserves image details compared to ST-LMS and THP & GM, as evidenced by visual inspection and the red rectangular areas highlighting preserved details in Figure 12.
  • Effective Ripple RNU Removal: The algorithm effectively removes ripple RNU, achieving a similar or better noise reduction capability than existing methods.
  • Faster Convergence: The method shows a faster convergence rate for ripple RNU correction, as indicated by the NU curves in Figure 16.
  • Reduced Ghosting Artifacts: The adaptive thresholding successfully suppresses ghosting artifacts, which are present in other methods (e.g., ST-LMS in Figure 12b).
  • High Scene Classification Accuracy: The fuzzy classifier achieved high accuracy for sky (98.9%) and ground (99.9%) scenes, with slightly lower accuracy for half-sky (98.3%), which is noted as a potential area for improvement.
  • FPGA Performance: The FPGA implementation offers low resource consumption (e.g., 1600 slices, 7 DSP48E) and a small hardware delay (<10 image rows), making it suitable for real-time processing.

Limitations and Future Work

  • Half-Sky Classification Accuracy: The paper acknowledges that the classification accuracy for half-sky images is the lowest. This can lead to misclassification, potentially causing detail loss in ground portions of half-sky images or insufficient RNU removal if misclassified as ground.
  • Parameter Tuning: The fuzzy classifier's performance relies on the correct setting of parameters like T1 and T2. The paper mentions that the thermal imager needs to be aimed at the sky at boot time for initial calculation of these parameters.
  • Future Work: The authors suggest future work could focus on "how to more accurately differentiate the ripple RNU from the scene," implying further refinement of the scene classification to better distinguish between true ripple RNU and other scene features, especially in complex half-sky scenarios.

Summary updated

Abstract

Cooled infrared detector arrays always suffer from undesired ripple residual nonuniformity (RNU) in sky scene observations. The ripple residual nonuniformity seriously affects the imaging quality, especially for small target detection. It is difficult to eliminate it using the calibration-based techniques and the current scene-based nonuniformity algorithms. In this paper, we present a modified temporal high-pass nonuniformity correction algorithm using fuzzy scene classification. The fuzzy scene classification is designed to control the correction threshold so that the algorithm can remove ripple RNU without degrading the scene details. We test the algorithm on a real infrared sequence by comparing it to several well-established methods. The result shows that the algorithm has obvious advantages compared with the tested methods in terms of detail conservation and convergence speed for ripple RNU correction. Furthermore, we display our architecture with a prototype built on a Xilinx Virtex-5 XC5VLX50T field-programmable gate array (FPGA), which has two advantages: (1) low resources consumption; and (2) small hardware delay (less than 10 image rows). It has been successfully applied in an actual system.
BibTeX
@inproceedings{li2017method,
  author = {Yiyang Li and Weiqi Jin and Shuo Li and Xu Zhang and Jin Zhu},
  title = {{A Method of Sky Ripple Residual Nonuniformity Reduction for a Cooled Infrared Imager and Hardware Implementation}},
  booktitle = {Sensors},
  year = {2017},
  doi = {10.3390/s17051070},
}
Figure