Object Detection in Hyperspectral Image via Unified Spectral-Spatial Feature Aggregation

Xiao He, Chang Tang, Xinwang Liu, Wei Zhang, Kun Sun, Jiangfeng Xu

Introduction

Hyperspectral imaging (HSI) provides a unique advantage over traditional RGB imaging by capturing data across hundreds of narrow spectral bands. This "spectral signature" allows for the differentiation of materials that may appear identical to the naked eye, such as distinguishing real vegetation from green plastic. However, the high dimensionality of HSI data creates a significant computational burden, often referred to as the "curse of dimensionality." Traditional object detection frameworks often struggle to balance the rich spectral data with spatial geometry, either ignoring the spectral depth or becoming overwhelmed by redundant information.

To address these challenges, this research introduces S2ADet, a novel detector designed to unify spectral and spatial feature aggregation, alongside HOD3K, a large-scale hyperspectral dataset to facilitate robust model training.

The S2ADet Framework

The S2ADet architecture is built on the principle that decoupling spectral and spatial information is as critical as their eventual fusion. The model consists of three primary stages:

1. Hyperspectral Information Decoupling (HID)

Rather than feeding raw hyperspectral cubes into a network, the HID module splits the data into two distinct streams. It utilizes Principal Component Analysis (PCA) to extract a Spectral (SE) stream and an Optimal Neighborhood Reconstruction (ONR) band selection method to create a Spatial (SA) stream. This process effectively removes spectral redundancy while preserving the most salient features for detection.

2. Two-Stream Feature Extraction

The model processes the SE and SA streams through a dual-backbone network based on DarkNet-FPN. This allows the system to learn specialized features for both the "what" (spectral characteristics) and the "where" (spatial shapes and boundaries) in parallel.

3. Spectral-Spatial Aggregation (SSA)

The SSA module serves as the core interaction layer. Using transformer-based attention mechanisms, the module allows the two streams to communicate, identifying complementary features between the spectral and spatial domains. This dynamic interaction ensures that the model can prioritize the most relevant spectral bands for specific spatial structures before the data reaches the final detection head.

The HOD3K Dataset

A significant contribution of this work is the release of the HOD3K dataset. Prior to this, the field was limited by small, specialized datasets. HOD3K contains 3,242 hyperspectral images (512×256 resolution) with 15,149 annotated objects across three classes: people, cars, and bikes. The dataset covers 16 bands ranging from 470 nm to 620 nm, providing a diverse training ground for deep learning models in HSI.

Experimental Results

S2ADet sets a new state-of-the-art (SOTA) benchmark for hyperspectral object detection:

  • HOD3K Performance: S2ADet achieved a 59.8 mAP (mean Average Precision), notably outperforming established models like Faster R-CNN (56.9) and YOLOv5 (54.4).
  • HOD-1 Performance: On the smaller HOD-1 benchmark, the model reached 86.6 mAP, a 3.1% improvement over previous leading methods.
  • Qualitative Advantages: The model demonstrates a superior ability to detect objects in "camouflaged" or high-occlusion scenes where standard RGB detectors often fail due to a lack of spectral contrast.

Conclusion and Future Directions

By explicitly decoupling and then intelligently aggregating spectral and spatial features, S2ADet overcomes the redundancy issues inherent in HSI data. While the model shows remarkable performance, challenges remain regarding the detection of extremely small or overlapping objects. Specifically, the "people" category in the HOD3K dataset remains difficult to distinguish from the background when targets are very small. Future work will focus on refining fine-grained classification and improving the handling of overlapping regions to further enhance detection accuracy in complex environments.

Summary updated

Abstract

Deep learning-based hyperspectral image (HSI) classification and object detection techniques have gained significant attention due to their vital role in image content analysis, interpretation, and broader HSI applications. However, current hyperspectral object detection approaches predominantly emphasize spectral or spatial information, overlooking the valuable complementary relationship between these two aspects. In this study, we present a novel spectral–spatial aggregation (S2ADet) object detector that effectively harnesses the rich spectral and spatial complementary information inherent in the HSI. S2ADet comprises a hyperspectral information decoupling (HID) module, a two-stream feature extraction network, and a one-stage detection head. The HID module processes hyperspectral data by aggregating spectral and spatial information via band selection and principal components analysis, consequently reducing redundancy. Based on the acquired spectral and spatial aggregation information, we propose a feature aggregation two-stream network for interacting spectral–spatial features. Furthermore, to address the limitations of existing databases, we annotate an extensive dataset, designated as HOD3K, containing 3242 HSIs captured across diverse real-world scenes, and encompassing three object classes. These images possess a resolution of $512\,\, {}\times {}256$ pixels and cover 16 bands ranging from 470 to 620 nm. Comprehensive experiments on two datasets demonstrate that S2ADet surpasses existing state-of-the-art methods, achieving robust, and reliable results. The demo code and dataset of this work are publicly available at https://github.com/hexiao-cs/S2ADet.
BibTeX
@misc{he2023object,
  author = {Xiao He and Chang Tang and Xinwang Liu and Wei Zhang and Kun Sun and Jiangfeng Xu},
  title = {{Object Detection in Hyperspectral Image via Unified Spectral-Spatial Feature Aggregation}},
  year = {2023},
  eprint = {2306.08370},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2306.08370},
}
Figure