Introduction
In the domain of long-range infrared imaging, the physical limitations of sensors—such as lens focal length and sensor resolution—often result in a phenomenon known as energy aliasing. When multiple targets are in close proximity, they frequently merge into a single, low-contrast "blob" in the image. Traditional Infrared Small Target Detection (IRSTD) methods typically operate under a one-to-one assumption: one identified spot corresponds to one physical target. However, this assumption fails in complex surveillance or early-warning scenarios where a single spot may mask a group of distinct objects.
This paper introduces the task of Sequential Closely-Spaced Infrared Small Target (SeqCSIST) Unmixing. Unlike standard detection, unmixing aims to identify the exact number and sub-pixel locations of multiple targets hidden within a single blurred region. To address this, the authors propose DeRefNet (Deformable Refinement Network), a model-driven architecture that leverages temporal information across video frames to achieve high-precision separation.
The One-to-Many Challenge
The core difficulty in CSIST unmixing lies in the "one-to-many" relationship. When targets are energy-aliased, their individual signatures are lost in the overlapping Point Spread Functions (PSF). Achieving sub-pixel precision in these dense scenarios is extremely difficult with standard deep learning architectures, which often treat the problem as a black-box regression task. Furthermore, the field has lacked high-quality, large-scale datasets specifically designed to benchmark the unmixing of closely-spaced targets in dynamic sequences.
Methodology: DeRefNet
DeRefNet distinguishes itself by combining mathematical optimization logic with the representational power of deep learning through three key components:
1. Deep Unfolding Backbone
Rather than using generic backbones like ResNet, DeRefNet employs a feature extraction module inspired by the Iterative Shrinkage-Thresholding Algorithm (ISTA). This "deep unfolding" approach integrates sparsity priors directly into the network architecture. By assuming that targets are small and sparse against the background, the model follows a mathematically sound optimization path, leading to faster convergence and greater robustness compared to purely heuristic models.
2. Temporal Deformable Feature Alignment (TDFA)
Temporal information serves as a critical constraint for resolving spatial ambiguity. While a single frame may be too blurred to distinguish two targets, their distinct trajectories over time provide the necessary geometric cues for separation. The TDFA module adaptively aligns multiple reference frames with a central frame. It utilizes a selective attention mechanism to focus on high-activation regions (the targets) while suppressing background noise and clutter.
3. Sub-Pixel Localization
To overcome the resolution limits of the original sensor, the network performs unmixing by upscaling feature maps early in the pipeline. This allows the model to predict target coordinates with a precision that exceeds the raw pixel grid, effectively "unmixing" the energy-aliased blobs into their constituent components.
The SeqCSIST Dataset
To facilitate research in this area, the authors developed the SeqCSIST benchmark, a large-scale sequential dataset consisting of 100,000 frames across 5,000 trajectories. The dataset simulates realistic optical degradation using PSF and Airy disk models to create authentic energy aliasing. Each sequence features 2 to 4 targets moving along various paths, including quadratic, circular, and straight trajectories.
To ensure real-world applicability, the authors also introduced a Hybrid Dataset. This was created by superimposing synthetic target trajectories onto 5,000 real-world infrared background sequences featuring diverse environments such as mountains, forests, and urban buildings.
Experimental Results
DeRefNet demonstrates significant improvements over existing state-of-the-art methods:
- Accuracy: The model achieved a 51.55% CSO-mAP (Closely-Spaced Object mean Average Precision), representing a 5.3% improvement over previous benchmarks like ISTA-Net++.
- Efficiency: Despite the complexity of sequential processing, DeRefNet operates at 367 FPS on standard hardware, far exceeding the requirements for real-time video processing (typically 25–30 FPS).
- Robustness: The model maintains high performance under sensor noise. In tests with moderate noise ($\sigma=2$), the mAP dropped by only approximately 2.5%.
- Generalization: Testing on the Hybrid Dataset showed only a 4% performance decrease compared to purely synthetic data, indicating that the model generalizes well to real-world infrared backgrounds.
Conclusion and Future Work
SeqCSIST unmixing represents a significant step forward in infrared target tracking, moving from simple detection to high-fidelity resolution of complex target clusters. By leveraging temporal constraints and model-driven deep learning, DeRefNet provides a robust solution for real-time sub-pixel localization.
The primary remaining challenge is the acquisition of real-world ground truth data with sub-pixel accuracy, which is logistically difficult to obtain. Future research will likely focus on domain adaptation techniques to further bridge the gap between synthetic training environments and purely real-world sensor data without the need for manual sub-pixel labeling.