TL;DR
This paper introduces a memory-efficient CNN hardware accelerator that compresses interlayer feature maps on-the-fly using a hardware-implemented Discrete Cosine Transform (DCT). By discarding high-frequency components and using sparse matrix encoding, the design significantly reduces on-chip memory requirements and off-chip memory access bandwidth for edge devices, achieving up to a 3.3x reduction in feature map size with less than 1% accuracy loss.
What Problem Does This Paper Address?
Deep Convolutional Neural Networks (CNNs) are highly accurate but generate massive amounts of intermediate data—known as interlayer feature maps—during inference. Storing and moving this data requires large on-chip SRAM or frequent, power-hungry off-chip DRAM accesses. This creates a severe "memory wall" bottleneck, especially for power-constrained mobile and IoT devices.
While many previous works have focused on compressing network weights or exploiting the sparsity created by ReLU activation functions (which turn negative values to zero), modern networks often use non-ReLU activations (like leaky-ReLU or Mish), resulting in dense feature maps. This paper tackles the challenge of compressing these dense, real-time feature maps directly in hardware without slowing down the computation pipeline or destroying model accuracy.
Methods
The authors propose a hardware accelerator that integrates compression and decompression directly into the CNN computing stream. The key techniques include:
- DCT-Based Frequency Compression: Treating feature maps like standard images, the accelerator applies an 8x8 Discrete Cosine Transform (DCT) to convert spatial data into the frequency domain.
- Layer-Aware Quantization: Similar to JPEG compression, a quantization table (Q-table) is used to discard high-frequency components (which usually represent noise or redundant data). The quantization aggressiveness is tuned per layer—early layers are compressed heavily, while deeper layers use finer quantization to preserve accuracy.
- Sparse Matrix Encoding with "Flip" Storage: After quantization, the resulting sparse matrices are encoded using a 1-bit index matrix (storing only non-zero values). To maximize SRAM utilization, the authors introduce a clever "flip" storage method: sequential 8x8 matrices are stored in reverse order so their empty spaces interlock perfectly in memory.
- Reconfigurable Memory Architecture: Because feature maps shrink in spatial size but grow in channel depth as the network progresses, the on-chip SRAM is dynamically reconfigurable. The boundary between the feature map buffer and the scratch pad (used for partial sums) shifts based on the specific needs of the current layer.
- Data MUX for Overlap: A specialized data multiplexer in the Processing Element (PE) array handles the overlapping data required for 3x3 convolutions, preventing the need to decompress the same feature map regions multiple times.
Datasets
The compression strategy and hardware performance were evaluated using the PASCAL VOC test dataset. The authors tested their approach on five standard pre-trained models: VGG-16-BN, ResNet-50, MobileNet-v1, MobileNet-v2, and Yolo-v3.
Key Insights and Ideas
- Feature Maps are Image-Like: The authors observed that the early layers of a CNN produce feature maps that retain strong spatial correlations and object shapes, much like the original input image. This makes them perfect candidates for frequency-domain compression techniques (like DCT) traditionally reserved for image and video files.
- On-the-Fly Processing Beats Off-Chip Storage: Moving data to off-chip DRAM is incredibly energy-intensive. By embedding DCT compression/decompression directly into the accelerator's pipeline, the system avoids massive latency and energy penalties. The power saved by avoiding DRAM accesses vastly outweighs the power consumed by the DCT hardware.
- Memory Flexibility is Crucial: A static memory allocation scheme wastes space. By allowing the SRAM to dynamically reallocate between feature map storage and partial sum accumulation, the hardware maintains high utilization across the radically different shapes of early vs. deep network layers.
Key Results
- Hardware Performance: Synthesized in TSMC 28nm HPC+ technology, the accelerator achieves a peak throughput of 403 GOPS with an energy efficiency of 2.16 TOPS/W at 700 MHz.
- Compression Ratios: The architecture achieved overall feature map compression ratios ranging from 30.63% (VGG-16-BN) to 71.05% (MobileNet-v2). This translates to a 1.4x to 3.3x reduction in off-chip memory access.
- Accuracy: Across all tested models, the DCT compression and quantization resulted in less than 1% accuracy loss.
- Hardware Overhead: The dedicated DCT/IDCT modules add only a 13% area overhead to the accelerator and consume 19% of the dynamic power (which is heavily offset by DRAM energy savings).
Limitations and Future Work
- Diminishing Returns on Deep/Lightweight Layers: The compression method is highly effective on the first 10-20 layers where spatial correlation is high. However, highly abstract deep layers, or lightweight networks that already use depth-wise convolutions (like MobileNet), are inherently harder to compress further.
- Fully Connected Layers Excluded: The proposed hardware focuses strictly on convolutional, pooling, and activation layers. Fully connected layers are offloaded to the CPU and fall outside the scope of this specific compression pipeline.
- Future Work: While not explicitly detailed, natural next steps would involve adapting this on-the-fly compression paradigm to handle the highly abstract feature maps of deeper layers or extending the architecture to support Transformer-based vision models.
Summary updated