TL;DR
This paper introduces a novel, training-free, one-shot method for segmenting small targets in sequential infrared images. By cleverly adapting the powerful Segment Anything Model (SAM) with custom modules, it achieves impressive accuracy using just a single annotated frame, overcoming the need for extensive training data and manual prompting for each frame.
What Problem Does This Paper Address?
Infrared small target segmentation (IRSTS) is crucial for applications like security and surveillance. While many existing methods focus on single images, sequential infrared data offers rich temporal context that can improve segmentation. However, effectively leveraging this sequential information is challenging. Traditional methods often require large annotated datasets for training, which is time-consuming and expensive. Furthermore, applying general-purpose segmentation models like SAM to IRSTS faces two main hurdles: the need for manual prompts for every frame (impractical for sequences) and a "domain gap" because SAM was primarily trained on RGB images, not infrared ones, leading to lower accuracy on small, dim targets. This paper aims to bridge this gap by enabling accurate IRSTS with minimal data and effort.
Methods
The proposed method is a three-stage, training-free pipeline that leverages SAM's generalization capabilities:
- Local Feature Matching (LFM): This module generates a "confidence map" by comparing features of a small target in a reference image with features from potential target regions in a test image. Instead of relying on manual prompts, the point with the highest confidence in this map is automatically selected as a prompt for SAM. This addresses the "frame-by-frame manual prompt" challenge.
- Point Prompt-Centric Focusing (PPCF): To combat SAM's tendency to over-segment or produce blurry masks on small infrared targets (due to the domain gap), this module focuses SAM's attention. It crops a local window around the automatically generated point prompt and feeds this focused region to SAM, effectively shifting from a global to a local view for more precise segmentation.
- Triple-Level Ensemble (TLE): Recognizing that window size significantly impacts feature matching and focusing, this module employs three different window sizes. It generates three preliminary segmentation masks and then combines them using a majority voting mechanism to produce the final, robust segmentation mask, mitigating misses and false detections.
Datasets
The experiments were conducted on the IRDST dataset, which contains 85 real-sequence scenarios for infrared small target detection.
Key Insights and Ideas
- Leveraging SAM's Generalization: The core idea is that SAM's powerful, pre-trained feature extractor can be adapted for IRSTS without further training, provided the input and prompting challenges are addressed.
- Automated Prompting: Replacing manual prompts with an automatically generated point prompt derived from local feature matching is a key innovation for practical sequential segmentation.
- Window Size Matters: The paper highlights that the size of the local window used for feature extraction and focusing significantly impacts segmentation accuracy. A larger window can include too much background, while a smaller one might miss subtle features. The TLE module directly addresses this by ensembling results from multiple window sizes.
- Domain Gap Mitigation: The PPCF module is crucial for adapting SAM, which is trained on RGB data, to the specific characteristics of infrared small targets.
Key Results
- The proposed one-shot method achieves performance comparable to state-of-the-art IRSTS models that are trained on large-scale datasets.
- It significantly outperforms other existing one-shot segmentation methods.
- Ablation studies confirm the effectiveness of each proposed module (PPCF and TLE).
- The method shows robustness to different types of annotations (point, bounding box, mask) and variations in the choice of the reference image.
- While increasing the number of shots (reference frames) can slightly improve detection probability (Pd), it negatively impacts the false alarm rate (Fa) and IoU.
Limitations and Future Work
- Limitations: The paper notes that while increasing the number of shots can improve Pd, it can negatively impact Fa and IoU. The "highest value point" from LFM might not always represent the target, though this is addressed by the subsequent modules.
- Future Work: Not explicitly detailed, but the robustness shown across different reference images and annotation types suggests potential for further exploration in few-shot and zero-shot scenarios with even less prior information. The method's reliance on SAM's architecture implies that future improvements could also stem from advancements in general-purpose segmentation models.
Summary updated