TL;DR
M4BRAM is a novel compute-in-memory (CIM) architecture that transforms FPGA Block RAMs into flexible, mixed-precision engines for deep learning. By allowing BRAMs to perform matrix multiplication while simultaneously serving data to DSP blocks, it achieves a 2.16× speedup over standard accelerators with negligible accuracy loss.
What Problem Does This Paper Address?
Deep Neural Networks (DNNs) are increasingly using mixed-precision quantization—where different layers or even different operands (weights vs. activations) use different bit-widths—to save power and memory. However, current FPGA architectures struggle to capitalize on this.
Standard Digital Signal Processing (DSP) blocks are designed for fixed, high-precision arithmetic; "packing" multiple low-precision operations into one DSP is often inefficient and doesn't scale linearly with lower precision. Previous attempts to solve this using Compute-In-Memory (CIM) in Block RAMs (BRAMs) had a major catch: they usually "locked" the BRAM, preventing it from acting as a normal memory buffer for the rest of the chip. This forced designers to choose between using BRAM for compute or using it to feed the DSPs, limiting the overall system's potential.
Methods
The authors propose M4BRAM, a modified BRAM architecture designed to complement, rather than replace, existing DSP resources.
Key technical components include:
- Decoupled Processing Elements (BPEs): Four small compute units are added to the BRAM periphery. These perform the actual math, leaving the main memory array mostly free.
- Single-Port CIM: Unlike prior work that occupied both BRAM ports for computation, M4BRAM uses only one port for CIM. This allows the second port to remain available for the DSP to read data simultaneously—enabling a "heterogeneous" compute flow.
- Duplication Shuffler: A novel circuit that can reorder and replicate weight vectors. This allows the hardware to exploit both weight-sharing and activation-sharing parallelism, which is critical for maintaining high utilization across different neural network topologies.
- Mixed-Precision Support: It supports 2, 4, or 8-bit weights and variable 2-to-8-bit activations. Because it uses a bit-serial approach for activations, the compute time scales linearly with precision (e.g., 4-bit activations take half the time of 8-bit).
Datasets
The researchers evaluated M4BRAM using the ImageNet classification task across several popular DNN architectures:
- AlexNet
- VGG-16
- ResNet-18 & ResNet-34
- ViT-Base (Vision Transformer)
They used these models to demonstrate that their quantization schemes maintain high accuracy while reaping significant hardware performance gains.
Key Insights and Ideas
- True Interoperability: The "aha moment" is that CIM shouldn't be an "either/or" proposition. By keeping one memory port open, M4BRAM acts as a "smart buffer" that computes partial results while feeding the main processor.
- Linear Performance Scaling: Most FPGA compute resources have "staircase" performance (e.g., you get the same speed for 5-bit as 8-bit). M4BRAM’s bit-serial activation processing ensures that every bit saved in quantization translates directly into a performance boost.
- Flexibility via Shuffling: Neural networks have diverse shapes. By adding the Duplication Shuffler, the authors allow the BRAM to adapt its internal parallelism ($N_W$ vs $N_I$) to match the specific needs of a layer, preventing hardware "idling."
Key Results
- Speedup: A heterogeneous accelerator (Hetero-DLA) using M4BRAM achieved an average 2.16× speedup over a baseline accelerator using standard BRAM.
- Efficiency vs. Prior Art: M4BRAM delivered 1.43× higher performance than BRAMAC (the previous state-of-the-art CIM architecture) because of its superior dataflow and DSP interoperability.
- Accuracy: The performance gains came with a negligible Top-1 accuracy loss of < 0.5% compared to full-precision floating-point models.
- Area Overhead: While M4BRAM increases the area of an individual BRAM block by 19-33%, it only increases the total FPGA core area by about 5.6% to 9.5%, a small price for doubling the throughput.
Limitations and Future Work
- Manual Tiling and Mapping: While the authors developed a design space exploration tool to find optimal tiling, fully automating this into a standard compiler (like TVM or Xilinx Vitis) remains a future challenge.
- Hardware Specifics: The study is modeled on Intel Stratix-10 (M20K BRAM). While the principles apply to Xilinx FPGAs, the specific area and frequency trade-offs might differ.
- Workload Partitioning: The current implementation partitions work primarily along the output feature dimension ($Q_{VEC}$). The authors note that exploring partitioning across other dimensions (like input channels) could unlock even more parallelism.
Summary updated