A2S-NAS: Asymmetric Spectral-Spatial Neural Architecture Search for Hyperspectral Image Classification

Lin Zhan, Jiayuan Fan, Peng Ye, Jianjian Cao

TL;DR

A2S-NAS is a new automated framework that designs custom neural networks for Hyperspectral Image (HSI) classification. By searching for the optimal combination of asymmetric pooling and multi-scale 3D convolutions, it outperforms hand-crafted and general-purpose architecture search methods on standard remote sensing benchmarks.

What Problem Does This Paper Address?

Hyperspectral images are unique: unlike a standard RGB photo with three color channels, an HSI contains hundreds of continuous spectral bands for every pixel. This creates a "spectral-spatial" cube that is rich in information but difficult to process.

The authors identify two major gaps in current deep learning approaches for HSI:

  1. Fixed Receptive Fields: Most networks use fixed-size convolution kernels. However, ground objects (like buildings vs. fields) come in vastly different sizes and shapes, requiring flexible "receptive fields" to be identified accurately.
  2. Asymmetric Dimensions: The spatial resolution (width/height) and spectral resolution (number of bands) of HSI data are physically different. Treating them symmetrically during pooling or convolution often leads to the loss of essential features.

Manually designing a network for every new HSI dataset is labor-intensive. While Neural Architecture Search (NAS) can automate this, existing NAS methods are often borrowed from general computer vision and don't account for the specific asymmetric nature of hyperspectral data.

Methods

The researchers propose A2S-NAS (Asymmetric Spectral-Spatial NAS), which uses a differentiable search strategy (similar to DARTS) to find the best architecture within a specialized search space.

The core of their method is the A2SConv block, which features a nested search structure:

  • Outer Search Space (Pooling): The model decides between three operations for each layer: no pooling, spectral pooling (reducing the number of bands), or spatial pooling (reducing the image dimensions). This allows the network to downsample the data asymmetrically based on where the most redundant information lies.
  • Inner Search Space (Convolution): Within each block, the model searches for the best 3D convolution parameters. It can choose between different kernel sizes ($3\times3\times3$ or $5\times5\times5$) and dilation rates. Dilation allows the network to expand its field of view without increasing the number of parameters.

To make the search stable and efficient, they employ Beta-Decay regularization, which helps the gradient-based search converge to a compact, high-performing final network without the massive computational cost of earlier NAS methods.

Datasets

The model was evaluated on two widely recognized HSI benchmarks:

  • Indian Pines (IP): A dataset primarily of agricultural land, containing 16 classes and 200 spectral bands.
  • Houston University (HU): An urban dataset with 15 classes (like trees, roads, and buildings) and 144 spectral bands.

Key Insights and Ideas

  • Asymmetry is a Feature, Not a Bug: The most striking finding is that the searched models heavily favored asymmetric pooling. In their final architectures, asymmetric operations accounted for 50% to 83.3% of the layers. This confirms that "one-size-fits-all" symmetric pooling is suboptimal for HSI.
  • Automated Flexibility: By allowing the network to choose its own receptive fields (via kernel size and dilation), the model became significantly better at identifying classes that vary in scale, such as "Roads" in urban environments or specific crop types in agricultural fields.
  • Efficiency via Differentiable Search: By using a continuous search space, the authors avoided the "hundreds of GPU days" typically associated with reinforcement learning-based NAS, making the design process accessible for practical remote sensing applications.

Key Results

A2S-NAS achieved state-of-the-art performance across both datasets, beating established models like SSRN and other NAS-based approaches like 3D-ANAS.

  • Indian Pines: Achieved an Overall Accuracy (OA) of 97.17%, notably improving accuracy for difficult classes like "Soybean-mintill" by over 3.5% compared to the next best model.
  • Houston University: Achieved an OA of 92.20%, with significant gains in identifying "Roads" (a 3.14% improvement).
  • Evaluation Metrics: The model consistently led in Overall Accuracy (OA), Average Accuracy (AA), and the Kappa coefficient, proving its robustness across different types of land cover.

Limitations and Future Work

While the paper demonstrates superior accuracy, it does not deeply explore the computational overhead (inference time or memory footprint) of the resulting 3D-heavy architectures compared to simpler 2D models.

Additionally, the search is currently conducted on a per-dataset basis. Future work could investigate Cross-Dataset NAS, where a single search process yields a "universal" HSI backbone that generalizes across different sensors and resolutions without needing to be re-searched from scratch.

Summary updated

Abstract

Existing deep learning-based hyperspectral image (HSI) classification works still suffer from the limitation of the fixed-sized receptive field, leading to difficulties in distinctive spectral-spatial features for ground objects with various sizes and arbitrary shapes. Meanwhile, plenty of previous works ignore asymmetric spectral-spatial dimensions in HSI. To address the above issues, we propose a multi-stage search architecture in order to overcome asymmetric spectral-spatial dimensions and capture significant features. First, the asymmetric pooling on the spectral-spatial dimension maximally retains the essential features of HSI. Then, the 3D convolution with a selectable range of receptive fields overcomes the constraints of fixed-sized convolution kernels. Finally, we extend these two searchable operations to different layers of each stage to build the final architecture. Extensive experiments are conducted on two challenging HSI benchmarks including Indian Pines and Houston University, and results demonstrate the effectiveness of the proposed method with superior performance compared with the related works.
BibTeX
@inproceedings{zhan2023s,
  author = {Lin Zhan and Jiayuan Fan and Peng Ye and Jianjian Cao},
  title = {{A2S-NAS: Asymmetric Spectral-Spatial Neural Architecture Search for Hyperspectral Image Classification}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=NyOYvtZLeh},
}

Loading related content...

Figure