Abstract
Convolutional neural networks (CNNs) are highly effective for hyperspectral imagery (HSI) classification, with 3D and hybrid 2D–3D models offering superior performance through joint spatial-spectral feature extraction. However, HSI classification presents unique computational and memory challenges compared to standard object detection, making conventional FPGA acceleration approaches less effective. We propose a dynamically configurable accelerator architecture that supports both 2D and 3D CNNs. By optimizing nested loops and parameterizing convolutional layers, our architecture facilitates rapid development and scaling across various networks. We introduce a parallelism-oriented memory pattern and data access strategy to optimize the data path. Implementing this on two FPGA platforms, we achieved a maximum throughput of 166.18 GOPS for HybridSN and a 10.7% power efficiency gain over baseline 3D models.
The Challenge of HSI Classification
Hyperspectral sensors capture hundreds of narrow spectral bands, creating a "data cube" rich in information. While 3D CNNs are ideal for this data, they are computationally expensive. Most existing FPGA accelerators are optimized for 2D CNNs used in standard RGB imagery. HSI classification is distinct: it typically processes very small spatial patches (e.g., 5x5 pixels) with high spectral depth. Standard acceleration often fails to utilize hardware resources efficiently for these specific data shapes. Furthermore, designing custom hardware for every new model is inefficient; there is a clear need for a reconfigurable "one-size-fits-many" architecture.
Proposed Configurable Architecture
The proposed architecture is implemented using RTL for fine-grained control and high resource utilization. It centers on a dynamic configurable design that allows the network to be "rebuilt" in hardware simply by changing parameters sent from an ARM processor, rather than rewriting hardware code.
Key technical components include:
- Parameterized Parallelism: By adjusting factors like $P_n, P_c, P_d$, the Processing Engine (PE) array scales to different FPGA sizes.
- Hybrid Support: The hardware treats 2D convolution as a special case of 3D convolution (kernel depth = 1), allowing hybrid models like HybridSN to run seamlessly.
- Weight Reuse: Given the high weight-to-data ratio in HSI models, the system uses circular buffers and prefetching to hide memory latency.
Parallelism-Oriented Memory Management
To handle the complex 4D and 5D tensors in 3D convolutions, we developed a specialized data-reshaping strategy. This strategy concentrates data into minimized blocks that stream directly into the PEs, significantly reducing on-chip SRAM requirements.
As shown in the reshaping strategy, input feature maps and weight blocks are reorganized to align with the parallel processing capabilities of the PE array. This ensures that the data path is optimized for the specific tiling and unrolling factors defined in the configuration.
Experimental Results
The accelerator was prototyped on the high-performance Alinx AXU15EG and the lower-resource Avnet Zedboard using three benchmark datasets: Pavia University, Salinas Valley, and Indian Pines.
- Throughput: The system achieved a maximum throughput of 166.18 GOPS for HybridSN. For a baseline C3D model, it reached 160.25 GOPS, outperforming previous dedicated 3D accelerators.
- Efficiency: The design achieved a 10.7% gain in power efficiency compared to existing solutions.
- Accuracy: Using 8-bit fixed-point quantization, the models maintained high classification accuracy (98.23% to 100% on Indian Pines), proving that low-precision hardware is sufficient for HSI tasks.
Key Insights and Future Work
A critical finding was the "Small Patch" challenge: in HSI, intermediate data is often less than or equal to the weights. This led us to prioritize weight reuse over feature map reuse. While 8-bit quantization is effective, future work will explore even lower bit-widths (binary/ternary networks) and weight pruning to further reduce the memory footprint of Fully Connected layers, which currently act as a major bottleneck in models like HybridSN.
Summary updated