Mitigating Memory Wall Effects in CNN Engines with On-the-Fly Weights Generation

Stylianos I. Venieris, Javier Fernández-Marqués, Nicholas D. Lane

Introduction

As Convolutional Neural Networks (CNNs) continue to grow in depth and complexity, their deployment on hardware accelerators faces a significant bottleneck known as the "Memory Wall." This phenomenon describes the widening performance gap between high-speed on-chip computation and the limited bandwidth of off-chip memory. In many FPGA-based Single Computation Engines (SCEs), the massive weight tensors required for deep layers often exceed on-chip storage, forcing the system to fetch data from external RAM. This results in processing elements (PEs) idling while waiting for data, leading to severe resource underutilization.

To address these challenges, this paper introduces unzipFPGA, a full-stack framework designed to bypass the memory wall by generating weights on-the-fly. By shifting the burden from memory communication to on-chip logic, unzipFPGA achieves significant improvements in performance density and energy efficiency.

The unzipFPGA Framework

The core innovation of unzipFPGA is the CNN-WGen (CNN Weight Generation) module. Rather than fetching full, high-precision weights from external memory, the system fetches compressed "Alpha" coefficients. These coefficients are used to reconstruct full weight tensors in real-time using Orthogonal Variable Spreading Factor (OVSF) codes—binary orthogonal basis vectors commonly used in telecommunications.

Tiled Weights Generation (TiWGen)

To ensure the framework is scalable across different CNN architectures, the authors developed TiWGen. This tiling strategy decouples the weight generation process from specific layer dimensions. By processing weights in discrete tiles, the hardware design remains consistent regardless of whether it is processing a shallow layer with few channels or a deep, dense layer. This modularity allows for a highly efficient and predictable hardware layout.

Input Selective Processing Elements

Beyond memory bandwidth, SCEs often suffer from internal fragmentation when a layer's shape does not perfectly align with the fixed hardware grid (e.g., a 128-PE array processing a 64-channel layer). unzipFPGA introduces Input Selective PEs, which incorporate lightweight switches to enable a "work-stealing" mechanism. If a PE is slated to be idle due to a shape mismatch, it can dynamically assist neighboring units by processing their data, thereby balancing the computational load and maximizing throughput.

Hardware-Aware Software Optimization

The framework includes a software-based autotuning tool that bridges the gap between model accuracy and hardware constraints. This tool performs a bottleneck analysis for every layer in the network:

  1. Memory-Bound Layers: If a layer is limited by memory bandwidth, the autotuner increases the OVSF ratio (effectively increasing weight precision). Because the engine is already waiting for data, this extra computation is essentially "free," improving model accuracy without adding to the total execution latency.
  2. Compute-Bound Layers: For layers limited by logic, the tool optimizes for speed, ensuring the weights are generated as efficiently as possible to keep the PEs saturated.

This "bottleneck-driven precision" ensures that the hardware's capabilities are fully exploited at every stage of the inference process.

Experimental Results

The unzipFPGA framework was evaluated using standard benchmarks, including ImageNet (ResNet18, 34, and 50), CIFAR-10, and SqueezeNet1.1. The results demonstrate a significant leap over state-of-the-art FPGA accelerators and mobile GPUs.

  • Performance Density: unzipFPGA achieved up to 3.94× higher performance density (measured in inferences per second per DSP) compared to established accelerators like Snowflake and xDNN.
  • Energy Efficiency: When compared to an NVIDIA Jetson TX2 GPU under similar power constraints, the system provided an average of 2.57× better performance-per-watt.
  • Throughput: For ResNet34, the framework outperformed standard Taylor-pruning methods by 80% at the same accuracy level in scenarios where memory bandwidth was highly constrained.
  • Utilization: The Input Selective PE design contributed up to a 20% performance boost on suboptimally mapped layers, effectively eliminating the idling issues common in traditional SCE architectures.

Conclusion and Future Work

unzipFPGA demonstrates that in the modern hardware landscape, computation is often "cheaper" than communication. By utilizing FPGA logic to calculate weights rather than moving them across a congested memory bus, the framework effectively mitigates the memory wall.

While the current implementation requires a specialized fine-tuning phase to learn OVSF coefficients and is currently optimized for power-of-two kernel lengths, it opens new doors for FPGA acceleration. Future research may explore multi-tenant environments, where on-the-fly weight generation could allow multiple CNN models to share a single FPGA with minimal memory contention, further pushing the boundaries of edge AI efficiency.

Summary updated

BibTeX
@article{venieris2023mitigating,
  author = {Stylianos I. Venieris and Javier Fernández-Marqués and Nicholas D. Lane},
  title = {{Mitigating Memory Wall Effects in CNN Engines with On-the-Fly Weights Generation}},
  journal = {ACM Trans. Design Autom. Electr. Syst.},
  year = {2023},
  doi = {10.1145/3611673},
}
Figure