TL;DR
MiM-ISTD introduces a nested "Mamba-in-Mamba" architecture to solve the high computational cost of infrared small target detection (ISTD). By combining an Outer Mamba block for global context and an Inner Mamba block for local details, it achieves state-of-the-art accuracy while running $8\times$ faster and using 62.2% less GPU memory than current transformer-based models on high-resolution images.
What Problem Does This Paper Address?
Infrared Small Target Detection (ISTD) is a crucial binary segmentation task used in remote sensing and military tracking systems to isolate tiny targets from background clutter. The task is notoriously difficult because targets are often just a few pixels wide, lack distinct structural features, and easily blend into complex background noise.
Existing deep learning approaches generally fall into two categories, both with significant trade-offs:
- CNN-based models excel at extracting local features but miss the broader global context, leading to high false-alarm rates when background elements mimic target signatures.
- Hybrid CNN-Transformer models successfully capture both local and global features but inherit the quadratic computational complexity ($O(N^2)$) of self-attention mechanisms.
As infrared sensors move toward higher resolutions, this quadratic bottleneck makes real-time deployment on edge devices impossible due to massive GPU memory and latency constraints.
Methods
To break the $O(N^2)$ bottleneck, the authors leverage State Space Models (SSMs)—specifically Mamba—which scale linearly ($O(N)$) with sequence length. However, standard visual Mamba blocks struggle to capture the ultra-fine local details required for tiny targets.
To fix this "locality defect," the authors propose Mamba-in-Mamba (MiM-ISTD), a hierarchical architecture that processes images at two distinct granularities:
- Outer Mamba Block: Divides the input image into patches termed "visual sentences" to model long-range global dependencies across the entire scene.
- Inner Mamba Block: Decomposes each visual sentence further into smaller sub-patches called "visual words." It models the local relationships within each sentence independently using a shared network, keeping parameter overhead minimal.
- 2D Selective Scan (SS2D): Because Mamba is inherently 1D, the model uses a quad-directional scanning strategy to map 2D spatial information into sequences without losing directional context.
The features from both the word and sentence streams are aggregated through a four-stage hierarchical encoder, passed through an up-sampling module, and fed into a plain ResNet-based decoder to generate the final target mask.
Datasets
The model was evaluated on two widely accepted public benchmarks:
- NUAA-SIRST: Contains 427 infrared images of varying sizes, resized to $512 \times 512$ for standard training and testing.
- IRSTD-1k: A more challenging dataset consisting of 1,000 real-world infrared images ($512 \times 512$) featuring richer, more complex background scenarios.
For both datasets, an 80/20 split was used for training and testing.
Key Insights and Ideas
- The Locality Defect of Mamba: While Mamba is highly efficient for long sequences, its standard formulation is suboptimal for tasks requiring extreme local precision (like finding a 3-pixel target in a massive image).
- Granularity Hierarchies: Borrowing concepts from Transformer-in-Transformer (TNT) architectures and applying them to SSMs allows the network to maintain its linear complexity advantage while recovering fine-grained spatial awareness.
- Weight Sharing for Efficiency: By sharing the Inner Mamba network across all visual sentences, the local feature extraction adds virtually zero computational or parameter bloat to the model.
Key Results
MiM-ISTD achieves a superior balance of accuracy and efficiency compared to current state-of-the-art (SOTA) models like TCI-Former and DNANet:
- Accuracy: It achieved the highest Intersection over Union (IoU) and Normalized IoU (nIoU) on both datasets (e.g., 80.92% IoU on NUAA-SIRST and 70.36% IoU on IRSTD-1k). Notably, it achieved a 100% Probability of Detection ($P_d$) on the NUAA-SIRST test set.
- Speed and Memory: When scaling up to high-resolution $2048 \times 2048$ images, MiM-ISTD runs $8\times$ faster than the leading SOTA method and slashes GPU memory consumption by 62.2%, demonstrating true viability for real-time edge deployment.
Limitations and Future Work
Not discussed in the paper.
Summary updated