Configurable 2D-3D CNNs Accelerator for FPGA-Based Hyperspectral Imagery Classification

Wenjing He, Yuesong Yang, Shaohui Mei, Jian Hu, Wanqiu Xu, Shiqi HaoIEEE JSTARS 2023 · OpenReview

Abstract

Convolutional neural networks (CNNs) are highly effective for hyperspectral imagery (HSI) classification, with 3D and hybrid 2D–3D models offering superior performance through joint spatial-spectral feature extraction. However, HSI classification presents unique computational and memory challenges compared to standard object detection, making conventional FPGA acceleration approaches less effective. We propose a dynamically configurable accelerator architecture that supports both 2D and 3D CNNs. By optimizing nested loops and parameterizing convolutional layers, our architecture facilitates rapid development and scaling across various networks. We introduce a parallelism-oriented memory pattern and data access strategy to optimize the data path. Implementing this on two FPGA platforms, we achieved a maximum throughput of 166.18 GOPS for HybridSN and a 10.7% power efficiency gain over baseline 3D models.

The Challenge of HSI Classification

Hyperspectral sensors capture hundreds of narrow spectral bands, creating a "data cube" rich in information. While 3D CNNs are ideal for this data, they are computationally expensive. Most existing FPGA accelerators are optimized for 2D CNNs used in standard RGB imagery. HSI classification is distinct: it typically processes very small spatial patches (e.g., 5x5 pixels) with high spectral depth. Standard acceleration often fails to utilize hardware resources efficiently for these specific data shapes. Furthermore, designing custom hardware for every new model is inefficient; there is a clear need for a reconfigurable "one-size-fits-many" architecture.

Proposed Configurable Architecture

The proposed architecture is implemented using RTL for fine-grained control and high resource utilization. It centers on a dynamic configurable design that allows the network to be "rebuilt" in hardware simply by changing parameters sent from an ARM processor, rather than rewriting hardware code.

System overview showing the configuration of patterns, kernels, and PE arrays for online classification.

Key technical components include:

  • Parameterized Parallelism: By adjusting factors like $P_n, P_c, P_d$, the Processing Engine (PE) array scales to different FPGA sizes.
  • Hybrid Support: The hardware treats 2D convolution as a special case of 3D convolution (kernel depth = 1), allowing hybrid models like HybridSN to run seamlessly.
  • Weight Reuse: Given the high weight-to-data ratio in HSI models, the system uses circular buffers and prefetching to hide memory latency.

Parallelism-Oriented Memory Management

To handle the complex 4D and 5D tensors in 3D convolutions, we developed a specialized data-reshaping strategy. This strategy concentrates data into minimized blocks that stream directly into the PEs, significantly reducing on-chip SRAM requirements.

Data reshaping strategy for input feature maps and weight blocks to optimize PE throughput.

As shown in the reshaping strategy, input feature maps and weight blocks are reorganized to align with the parallel processing capabilities of the PE array. This ensures that the data path is optimized for the specific tiling and unrolling factors defined in the configuration.

Experimental Results

The accelerator was prototyped on the high-performance Alinx AXU15EG and the lower-resource Avnet Zedboard using three benchmark datasets: Pavia University, Salinas Valley, and Indian Pines.

  • Throughput: The system achieved a maximum throughput of 166.18 GOPS for HybridSN. For a baseline C3D model, it reached 160.25 GOPS, outperforming previous dedicated 3D accelerators.
  • Efficiency: The design achieved a 10.7% gain in power efficiency compared to existing solutions.
  • Accuracy: Using 8-bit fixed-point quantization, the models maintained high classification accuracy (98.23% to 100% on Indian Pines), proving that low-precision hardware is sufficient for HSI tasks.

Key Insights and Future Work

A critical finding was the "Small Patch" challenge: in HSI, intermediate data is often less than or equal to the weights. This led us to prioritize weight reuse over feature map reuse. While 8-bit quantization is effective, future work will explore even lower bit-widths (binary/ternary networks) and weight pruning to further reduce the memory footprint of Fully Connected layers, which currently act as a major bottleneck in models like HybridSN.

Summary updated

Abstract

Convolutional neural network (CNN) is used efficiently for the classification of hyperspectral imagery (HSI). Both 3-D CNN and hybrid 2D–3D CNN have better performance due to the full extraction of joint spatial-spectral features. Most of previous acceleration research on field-programmable gate array (FPGA) has concentrated on 2-D CNN, few of which were made for 3-D CNN acceleration. More importantly, it is notable that the CNN models for HSI classification have some unique characteristics of computation and memory, different from those for object detection. Thus, the conventional accelerating approaches may be not applicable. To address this problem, we propose a dynamic configurable accelerator architecture suitable for both 2-D and 3-D CNN, ensuring fast development for various networks. The multiple nested loops are optimized delicately and the convolutional layers are fully parameterized, making it easy to scale up the network. Furthermore, we develop the parallelism-oriented memory pattern and data access strategy to optimize the data path and the local buffer. Finally, we implement the proposed architecture for 2-D, 3-D, and hybrid CNNs, validating its effectiveness and reconfigurability. To demonstrate its extensibility, we also prototype the accelerator on two FPGA platforms. We achieve the average and maximum throughput of 18.2/166.18 giga operations per second (GOPS) for HybridSN. To compare with previous FPGA accelerators, a typical baseline 3-D model is also implemented with the average and maximum throughput 160.25/225.87 GOPS, and the performance efficiency achieves up to 1.23 with 10.7% power efficiency gain.
BibTeX
@article{he2023configurable,
  author = {Wenjing He and Yuesong Yang and Shaohui Mei and Jian Hu and Wanqiu Xu and Shiqi Hao},
  title = {{Configurable 2D-3D CNNs Accelerator for FPGA-Based Hyperspectral Imagery Classification}},
  journal = {IEEE J. Sel. Top. Appl. Earth Obs. Remote. Sens.},
  year = {2023},
  url = {https://openreview.net/forum?id=jozXOsRFrc},
}
Figure