Background Semantics Matter: Cross-Task Feature Exchange Network for Clustered Infrared Small Target Detection With Sky-Annotated Dataset

Yimian Dai, Mengxuan Xiao, Yiming Zhu, Huan Wang, Kehua Guo, Jian YangarXiv 2024 · DOI · arXiv

Introduction

Infrared small target detection (ISTD) is a critical task in computer vision, often described as finding a "needle in a haystack." In these scenarios, targets typically occupy only a few pixels and lack distinct textures, making them nearly indistinguishable from sensor noise or background "distractors" like building edges or sun glints. While traditional research has focused on sparse scenarios—one or two targets against a uniform sky—modern applications like drone swarm defense and maritime rescue demand the detection of clustered targets in complex, cluttered environments.

The Challenge of Clustered Targets

In dense scenes, traditional detection methods often fail. These methods typically rely on local contrast; however, when targets are bunched together, that contrast disappears. Furthermore, without environmental context, a bright spot on a building can look identical to a target in the sky.

Figure 1: The transition from local to global context. Local patches (top left) show the ambiguity between true targets (blue) and false alarms (pink). Global context (right and bottom) helps resolve these ambiguities by identifying background semantics.

The authors of this paper posit that the missing link in ISTD is background semantics. Knowing whether a bright pixel is located in the "sky" versus on a "building" serves as the ultimate tie-breaker for reducing false alarms.

BAFE-Net: A Background-Aware Approach

To address these challenges, the authors propose the Background-Aware Feature Exchange Network (BAFE-Net). The architecture is designed to treat the background not as noise to be suppressed, but as a semantic anchor.

Key Architectural Pillars

  1. Multi-Task Learning: BAFE-Net features two parallel branches: one for object detection (based on FCOS) and one for background semantic segmentation (e.g., sky vs. non-sky).
  2. Dynamic Channel Selection (DCS): Recognizing that not all features are equally useful for both tasks, the DCS module uses an attention mechanism to identify the "top-k" most discriminative channels for detection and segmentation at any given moment.
  3. Cross-Task Feature Hard-Exchange: Unlike standard "soft" fusion (like adding or concatenating features), which can dilute the tiny signal of a small target, BAFE-Net performs a "hard exchange." It swaps the most important channels between the two branches, allowing the detection branch to "borrow" global environmental context to suppress false positives.
  4. BAG-CP Augmentation: A "Background-Aware Gaussian Copy-Paste" strategy intelligently pastes synthetic targets into semantically appropriate regions with Gaussian blurring, ensuring the model learns realistic spatial relationships.

DenseSIRST: A New Benchmark

A significant contribution of this work is the DenseSIRST dataset. Containing 1,024 images and 13,655 targets, it is the first dataset to feature high-density clusters (averaging 13 targets per image) and provide per-pixel semantic labels for the background. This allows models to be trained and evaluated specifically on their ability to handle complex, crowded scenes.

Key Insights and Results

The research yielded several critical insights:

  • The Failure of Deep Backbones: Interestingly, deeper networks (like ResNet-101) often underperformed compared to shallower ones (ResNet-18 or ResNet-50). The extreme abstraction in very deep networks can "wash out" the tiny, single-pixel features essential for small target detection.
  • Hard-Exchange vs. Soft Fusion: By physically replacing channels rather than blending them, the network maintains "feature integrity," preventing the gradient conflicts often found in multi-task learning.

BAFE-Net achieved state-of-the-art performance, reaching a 0.274 mAP₀₇ on DenseSIRST. Qualitative results demonstrated significantly "cleaner" heatmaps with fewer false positives in complex background areas compared to standard one-stage detectors.

Conclusion and Future Work

BAFE-Net demonstrates that background semantics are vital for robust infrared small target detection. By integrating background segmentation and target detection through a hard-exchange mechanism, the model effectively utilizes global context to resolve local ambiguities. Future work may involve increasing semantic granularity (e.g., distinguishing clouds, sea, and mountains) and incorporating temporal context from video sequences to further improve tracking in highly cluttered environments.

Summary updated

BibTeX
@misc{dai2024background,
  author = {Yimian Dai and Mengxuan Xiao and Yiming Zhu and Huan Wang and Kehua Guo and Jian Yang},
  title = {{Background Semantics Matter: Cross-Task Feature Exchange Network for Clustered Infrared Small Target Detection With Sky-Annotated Dataset}},
  year = {2024},
  eprint = {2407.20078},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.20078},
}
Figure