Memory-Efficient CNN Accelerator Based on Interlayer Feature Map Compression

Zhuang Shao, Xiaoliang Chen, Li Du, Lei Chen, Yuan Du, Wei Zhuang, Huadong Wei, Chenjia Xie, Zhongfeng Wang

Abstract

Existing deep convolutional neural networks (CNNs) generate massive interlayer feature data during network inference. To maintain real-time processing in embedded systems, large on-chip memory is required to buffer the interlayer feature maps. In this paper, we propose an efficient hardware accelerator with an interlayer feature compression technique to significantly reduce the required on-chip memory size and off-chip memory access bandwidth. The accelerator compresses interlayer feature maps through transforming the stored data into frequency domain using hardware-implemented 8x8 discrete cosine transform (DCT). The high-frequency components are removed after the DCT through quantization. Sparse matrix compression is utilized to further compress the interlayer feature maps. The on-chip memory allocation scheme is designed to support dynamic configuration of the feature map buffer size and scratch pad size according to different network-layer requirements. The hardware accelerator combines compression, decompression, and CNN acceleration into one computing stream, achieving minimal compressing and processing delay. A prototype accelerator is implemented on an FPGA platform and also synthesized in TSMC 28-nm COMS technology. It achieves 403GOPS peak throughput and 1.4x~3.3x interlayer feature map reduction by adding light hardware area overhead, making it a promising hardware accelerator for intelligent IoT devices.

Introduction

Deep Convolutional Neural Networks (CNNs) have become the standard for high-accuracy computer vision tasks. However, their deployment on edge and IoT devices is hindered by the massive volume of intermediate data—interlayer feature maps—generated during inference. Storing and moving this data creates a "memory wall" bottleneck. While previous research has successfully compressed network weights or exploited sparsity in ReLU-based networks, modern architectures often use non-ReLU activations (like Leaky-ReLU or Mish), resulting in dense feature maps that require significant on-chip SRAM or power-hungry off-chip DRAM accesses.

This paper addresses this challenge by introducing a hardware accelerator that compresses these dense feature maps in real-time, reducing memory requirements without compromising the computational pipeline or model accuracy.

Methods

The proposed architecture integrates a compression and decompression engine directly into the CNN processing stream. This design minimizes latency by avoiding separate processing steps for data reduction.

The proposed compression and decompression pipeline integrated into the CNN hardware accelerator.

Key technical components include:

  • DCT-Based Frequency Compression: The accelerator treats feature maps as images, applying an 8x8 Discrete Cosine Transform (DCT) to shift spatial data into the frequency domain.
  • Layer-Aware Quantization: Similar to JPEG compression, a quantization table (Q-table) discards high-frequency components. Quantization is tuned per layer: early layers undergo aggressive compression, while deeper layers use finer quantization to preserve critical abstract features.
  • Sparse Matrix Encoding: Post-quantization, the resulting sparse matrices are stored using a 1-bit index matrix. A "flip" storage method is employed to interlock sequential matrices in memory, maximizing SRAM utilization.
  • Reconfigurable Memory Architecture: To accommodate the changing dimensions of feature maps (which shrink spatially but grow in channel depth), the on-chip SRAM dynamically reallocates space between the feature map buffer and the scratch pad used for partial sums.

Key Insights: Feature Maps as Images

A core observation driving this work is that early-layer feature maps retain strong spatial correlations and recognizable object shapes, behaving much like the original input images. This characteristic makes them ideal candidates for frequency-domain compression techniques like DCT.

Visualization of feature maps at different layers, showing image-like spatial correlations in early layers.

By embedding compression directly into the hardware pipeline, the system avoids the energy penalty of DRAM access. The power consumed by the DCT hardware is significantly lower than the energy required to move uncompressed data to off-chip memory.

Key Results

Evaluated using the PASCAL VOC dataset across models like VGG-16, ResNet-50, and YOLO-v3, the accelerator demonstrated significant efficiency gains:

  • Throughput: Synthesized in TSMC 28nm technology, it achieves a peak throughput of 403 GOPS and an energy efficiency of 2.16 TOPS/W.
  • Compression Efficiency: The architecture achieved feature map compression ratios between 30.63% and 71.05%, representing a 1.4x to 3.3x reduction in off-chip memory bandwidth.
  • Accuracy Preservation: Across all benchmarks, the compression and quantization resulted in less than 1% accuracy loss.
  • Overhead: The dedicated DCT/IDCT modules added only a 13% area overhead, a cost-effective trade-off for the resulting memory and power savings.

Conclusion

The proposed accelerator effectively bridges the gap between high-performance CNNs and resource-constrained edge devices. By leveraging the image-like properties of interlayer feature maps through hardware-accelerated DCT and quantization, the design significantly mitigates the memory bottleneck. Future work may explore extending this frequency-domain approach to deeper, more abstract layers or adapting the architecture for Transformer-based vision models.

Summary updated

Abstract

Existing deep convolutional neural networks (CNNs) generate massive interlayer feature data during network inference. To maintain real-time processing in embedded systems, large on-chip memory is required to buffer the interlayer feature maps. In this paper, we propose an efficient hardware accelerator with an interlayer feature compression technique to significantly reduce the required on-chip memory size and off-chip memory access bandwidth. The accelerator compresses interlayer feature maps through transforming the stored data into frequency domain using hardware-implemented 8x8 discrete cosine transform (DCT). The high-frequency components are removed after the DCT through quantization. Sparse matrix compression is utilized to further compress the interlayer feature maps. The on-chip memory allocation scheme is designed to support dynamic configuration of the feature map buffer size and scratch pad size according to different network-layer requirements. The hardware accelerator combines compression, decompression, and CNN acceleration into one computing stream, achieving minimal compressing and processing delay. A prototype accelerator is implemented on an FPGA platform and also synthesized in TSMC 28-nm COMS technology. It achieves 403GOPS peak throughput and 1.4x~3.3x interlayer feature map reduction by adding light hardware area overhead, making it a promising hardware accelerator for intelligent IoT devices.
BibTeX
@article{shao2022memory,
  author = {Zhuang Shao and Xiaoliang Chen and Li Du and Lei Chen and Yuan Du and Wei Zhuang and Huadong Wei and Chenjia Xie and Zhongfeng Wang},
  title = {{Memory-Efficient CNN Accelerator Based on Interlayer Feature Map Compression}},
  journal = {IEEE Trans. Circuits Syst. I Regul. Pap.},
  year = {2022},
  url = {https://openreview.net/forum?id=iBFYqOTzPv},
}
Figure