Small and Dim Target Detection in IR Imagery: A Review

Nikhil Kumar, Pravendra SinghSSRN 2023 · DOI · OpenReview

TL;DR

This paper provides a comprehensive review of the state-of-the-art in detecting small and dim targets within infrared (IR) imagery. It categorizes existing methodologies, from traditional image processing to cutting-edge deep learning, and highlights the challenges, datasets, and performance metrics crucial for this complex task. The review aims to guide future research by identifying gaps and limitations in current approaches.

What Problem Does This Paper Address?

Detecting small and dim targets in infrared (IR) imagery is a critical but challenging task with significant implications for applications like surveillance, reconnaissance, and search and rescue. These targets are difficult to detect due to their low thermal contrast against cluttered backgrounds, noise, and varying environmental conditions. Traditional optical sensors often fall short in these scenarios, making IR imagery essential. However, existing object detection algorithms, even advanced deep learning models, struggle when directly applied to IR imagery, often losing crucial information about these tiny targets in deeper network layers. This paper addresses the need for a consolidated understanding of the diverse techniques developed to tackle this specific problem.

Methods

The paper categorizes approaches into two main types: Multiple Frame Infrared Small Target Detection (MIRST) and Single Frame Infrared Small Target Detection (SIRST).

  • MIRST methods utilize multiple frames, often employing conventional image processing techniques like 3D matched filtering and regularity flow-based approaches. These methods can be computationally expensive and require image registration if the camera is moving.
  • SIRST methods are further divided into:
  • Conventional Image Processing Based Approaches: These include:
  • Low-rank Representation Based Approaches: Utilizing mathematical properties of matrices to represent images in a lower-dimensional space (e.g., IR Patch-Image model, Reweighted IR Patch-Tensor model).
  • Human Visual System (HVS) Based Approaches: Mimicking human visual perception to extract relevant information (e.g., Local Contrast Method).
  • Filtering Based Approaches: Employing techniques like Top Hat Morphology (THM), Modified Top Hat Morphology (MTHM), Contour Morphology (CM), and Method of Directional Derivatives (MODD) to enhance target features and suppress background noise.
  • Deep Learning Based Approaches: These are further classified into:
  • Fully Supervised Learning Based Approaches: * Detection Type: Modifying generic deep learning frameworks for small target detection (e.g., YOLO-FR, GANet).
  • Segmentation Type: Focusing on pixel-wise classification to distinguish targets from background (e.g., ACMnet, ALCNet, DNA-Net, ISNet, MDvsFA). These often involve sophisticated architectures like U-Nets, attention mechanisms, and feature fusion techniques.
  • Weakly Supervised Learning Based Approaches: Utilizing less demanding supervision, such as point-level labels, to reduce annotation costs (e.g., Label Evolution with Single Point Supervision - LESPS).

Datasets

The paper highlights the importance of datasets and lists several key ones:

  • NUAA-SIRST: Real IR images with diverse backgrounds (clouds, city, sea), annotated with coarse labels.
  • NUST-SIRST: Artificially generated synthetic images with diverse backgrounds, classified into point and spot types.
  • CQU-SIRST: Synthetic images focusing on point targets.
  • NUDT-SIRST: Synthetically generated images with a mix of point and extended targets.
  • IRSTD-1K: Real-world IR images captured by an IR camera, featuring significant clutter and noise, suitable for benchmarking.
  • Custom Datasets: Researchers often create their own datasets, sometimes combining synthetic backgrounds with synthetic targets.

Key Insights and Ideas

  • Deep Learning Dominance: Deep learning approaches consistently outperform traditional image processing methods, especially in complex and cluttered IR imagery.
  • The "Smallness" Problem: Standard CNN architectures, particularly their pooling layers, are detrimental to small target detection as they can discard crucial fine-grained information. This necessitates specialized architectures.
  • Data is King (But Expensive): Fully supervised deep learning methods require large, meticulously annotated datasets, which are costly and time-consuming to create. Weakly supervised methods offer a promising alternative by reducing annotation burden.
  • Beyond Generic Object Detection: Models designed for general object detection often fail when directly applied to IR small targets. Tailored architectures that consider the unique characteristics of IR imagery and small targets are crucial.
  • Fusion is Key: Combining information from multiple frames (MIRST) or fusing features from different network layers (deep learning) often leads to improved detection performance.
  • The Trade-off: There's an inherent trade-off between detection rate (DR) and false alarm rate (FAR) in many methods, requiring careful selection based on application priorities.

Key Results

  • The review indicates that deep learning approaches generally achieve superior performance compared to traditional methods.
  • Specific deep learning models like DNA-Net and ACMnet show promising results in segmentation-based detection, achieving high detection rates (e.g., Pd > 96% for DNA-Net on NUDT-SIRST).
  • Filtering-based methods like MTHM demonstrate a low FAR, though sometimes at the cost of a lower DR.
  • The paper provides comparative tables (e.g., Table 5 for deep learning methods) showcasing performance metrics like IoU, Precision, Recall, Pd, Fa, and mAP across various datasets and methods.

Limitations and Future Work

  • Hyperparameter Tuning: Many conventional methods require extensive hyperparameter tuning, making them challenging for automated systems.
  • Generalizability: Achieving robust performance across all scenarios with a single set of hyperparameters remains a challenge.
  • Dataset Limitations: While progress has been made, the availability of large-scale, diverse, and accurately annotated IR datasets is still a bottleneck.
  • Future Directions:
    • Development of advanced deep learning architectures specifically for IR small target detection.
  • Exploration of unsupervised and semi-supervised learning techniques.
  • Integration of multispectral and hyperspectral data for richer scene context.
  • Advancements in sensor technology for improved sensitivity and dynamic range.
  • Focus on real-time processing capabilities using parallel processing and hardware acceleration.

Summary updated

Abstract

While there has been significant progress in object detection using conventional image processing and machine learning algorithms, exploring small and dim target detection in the IR domain is a relatively new area of study. The majority of small and dim target detection methods are derived from conventional object detection algorithms, albeit with some alterations. The task of detecting small and dim targets in IR imagery is complex. This is because these targets often need distinct features, the background is cluttered with unclear details, and the IR signatures of the scene can change over time due to fluctuations in thermodynamics. The primary objective of this review is to highlight the progress made in this field. This is the first review in the field of small and dim target detection in infrared imagery, encompassing various methodologies ranging from conventional image processing to cutting-edge deep learning-based approaches. The authors have also introduced a taxonomy of such approaches. There are two main types of approaches: methodologies using several frames for detection, and single-frame-based detection techniques. Single frame-based detection techniques encompass a diverse range of methods, spanning from traditional image processing-based approaches to more advanced deep learning methodologies. Our findings indicate that deep learning approaches perform better than traditional image processing-based approaches. In addition, a comprehensive compilation of various available datasets has also been provided. Furthermore, this review identifies the gaps and limitations in existing techniques, paving the way for future research and development in this area.
BibTeX
@misc{kumar2023small,
  author = {Nikhil Kumar and Pravendra Singh},
  title = {{Small and Dim Target Detection in IR Imagery: A Review}},
  year = {2023},
  eprint = {2311.16346},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2311.16346},
  howpublished = {SSRN},
}
Figure