Spectral Discrepancy and Cross-Modal Semantic Consistency Learning for Object Detection in Hyperspectral Images

Xiao He, Chang Tang, Xinwang Liu, Wei Zhang, Zhimin Gao, Chuankun Li, Shaohua Qiu, Jiangfeng Xu

Introduction

Hyperspectral imaging (HSI) represents a significant advancement over traditional RGB imaging by capturing dozens or even hundreds of narrow spectral bands across the electromagnetic spectrum. This high-dimensional data provides a unique "spectral signature" for every pixel, allowing for the identification of materials that appear identical to the human eye—such as distinguishing real vegetation from green plastic camouflage. However, the transition from three-channel RGB to high-dimensional HSI introduces two primary challenges: inter-band heterogeneity and data redundancy.

Inter-band heterogeneity occurs because bands at different ends of the spectrum (e.g., visible vs. near-infrared) can represent the same scene with vastly different intensities and features, often leading to conflicting information in standard neural networks. Simultaneously, the sheer volume of HSI data contains significant redundancy, where adjacent bands provide little new information while increasing computational load and noise. This paper introduces the Spectral Discrepancy and Cross-modal semantic consistency learning (SDCM) framework to address these hurdles by treating spectral bands as distinct modalities and employing intelligent filtering mechanisms.

The SDCM Framework

The SDCM framework is built upon three core innovations designed to align spectral information and prune irrelevant data:

1. Semantic Consistency Learning (SCL)

Rather than treating hyperspectral bands as a simple monolithic stack, the SCL module treats different spectral ranges as distinct "modalities," similar to how a model might process a combination of RGB and thermal data. It utilizes a dynamic cross-modal spatial attention mechanism to identify and extract a unified representation. This ensures that features extracted from disparate wavelengths are semantically aligned, resolving the conflicts inherent in inter-band heterogeneity.

2. Spectral Gated Generator (SGG)

To combat data redundancy, the SGG acts as an intelligent filter. It employs a parameter-free 3D attention mechanism to calculate an "energy function" for each neuron, effectively measuring its information density. By suppressing redundant or noisy spectral content and focusing the model’s resources on the most informative bands, the SGG improves both the accuracy and the efficiency of spatial localization.

3. Spectral Discrepancy Aware (SDA) Module

The SDA module focuses on high-level semantic representation. It utilizes a spectral filter (such as Principal Component Analysis) to extract pixel-level spectral signatures. These signatures are then processed through a Transformer-based decoder to refine final object predictions, ensuring that the model leverages the unique spectral "discrepancy" of objects relative to their backgrounds.

Experimental Evaluation

The SDCM framework was evaluated on two distinct and challenging benchmarks:

  • HOD-1: A dataset specifically curated for camouflage detection. It consists of 1,657 images across 96 spectral bands, featuring objects meticulously hidden against backgrounds with nearly identical spatial textures.
  • HOD3K: A larger-scale dataset containing 3,242 images and 16 spectral bands. This dataset focuses on traditional remote sensing categories, such as pedestrians, bicycles, and vehicles in residential and roadway environments.

Results and Key Insights

The SDCM framework achieved state-of-the-art (SOTA) performance across both datasets, demonstrating the efficacy of the cross-modal approach.

  • HOD-1 Performance: SDCM reached a 93.6% [email protected], a substantial 10% improvement over the previous leading method, S2ADet. The model was particularly successful in difficult categories like "pen screen," where accuracy more than doubled compared to baseline detectors.
  • HOD3K Performance: The model achieved 95.2% [email protected], significantly outperforming established general-purpose detectors such as Faster R-CNN and YOLOv5.
  • Ablation Studies: Analysis revealed that the SCL module alone contributed an 8.9% boost in mAP. This confirms that resolving inter-band inconsistency is the most critical factor in improving HSI object detection.

Discussion and Conclusion

The success of SDCM highlights a paradigm shift in hyperspectral processing: viewing spectral bands as distinct modalities rather than just additional channels. By explicitly modeling the consistency between these bands and pruning the spectral search space via the SGG, the model can "see through" camouflage that is spatially and texturally perfect.

Despite these advancements, the authors note that challenges remain regarding small and heavily occluded objects, which still suffer from background clutter. Furthermore, while the SGG filters data, the multi-modal attention mechanisms remain computationally intensive. Future research will focus on optimizing these architectures for real-time applications and exploring more potent methods for integrating diverse wavelength bands into a unified conduit for environmental and architectural monitoring.

Summary updated

Abstract

Hyperspectral images with high spectral resolution provide new insights into recognizing subtle differences in similar substances. However, object detection in hyperspectral images faces significant challenges in intra- and inter-class similarity due to the spatial differences in hyperspectral inter-bands and unavoidable interferences, e.g., sensor noises and illumination. To alleviate the hyperspectral inter-bands inconsistencies and redundancy, we propose a novel network termed \textbf{S}pectral \textbf{D}iscrepancy and \textbf{C}ross-\textbf{M}odal semantic consistency learning (SDCM), which facilitates the extraction of consistent information across a wide range of hyperspectral bands while utilizing the spectral dimension to pinpoint regions of interest. Specifically, we leverage a semantic consistency learning (SCL) module that utilizes inter-band contextual cues to diminish the heterogeneity of information among bands, yielding highly coherent spectral dimension representations. On the other hand, we incorporate a spectral gated generator (SGG) into the framework that filters out the redundant data inherent in hyperspectral information based on the importance of the bands. Then, we design the spectral discrepancy aware (SDA) module to enrich the semantic representation of high-level information by extracting pixel-level spectral features. Extensive experiments on two hyperspectral datasets demonstrate that our proposed method achieves state-of-the-art performance when compared with other ones.
BibTeX
@article{he2025spectral,
  author = {Xiao He and Chang Tang and Xinwang Liu and Wei Zhang and Zhimin Gao and Chuankun Li and Shaohua Qiu and Jiangfeng Xu},
  title = {{Spectral Discrepancy and Cross-Modal Semantic Consistency Learning for Object Detection in Hyperspectral Images}},
  journal = {IEEE Trans. Multimedia},
  year = {2025},
  doi = {10.1109/tmm.2025.3586155},
}
Figure