M4BRAM: Mixed-Precision Matrix-Matrix Multiplication in FPGA Block RAMs

Yuzong Chen, Jordan Dotzel, Mohamed S. Abdelfattah

TL;DR

M4BRAM is a novel compute-in-memory (CIM) architecture that transforms FPGA Block RAMs into flexible, mixed-precision engines for deep learning. By allowing BRAMs to perform matrix multiplication while simultaneously serving data to DSP blocks, it achieves a 2.16× speedup over standard accelerators with negligible accuracy loss.

What Problem Does This Paper Address?

Deep Neural Networks (DNNs) are increasingly using mixed-precision quantization—where different layers or even different operands (weights vs. activations) use different bit-widths—to save power and memory. However, current FPGA architectures struggle to capitalize on this.

Standard Digital Signal Processing (DSP) blocks are designed for fixed, high-precision arithmetic; "packing" multiple low-precision operations into one DSP is often inefficient and doesn't scale linearly with lower precision. Previous attempts to solve this using Compute-In-Memory (CIM) in Block RAMs (BRAMs) had a major catch: they usually "locked" the BRAM, preventing it from acting as a normal memory buffer for the rest of the chip. This forced designers to choose between using BRAM for compute or using it to feed the DSPs, limiting the overall system's potential.

Methods

The authors propose M4BRAM, a modified BRAM architecture designed to complement, rather than replace, existing DSP resources.

Key technical components include:

  • Decoupled Processing Elements (BPEs): Four small compute units are added to the BRAM periphery. These perform the actual math, leaving the main memory array mostly free.
  • Single-Port CIM: Unlike prior work that occupied both BRAM ports for computation, M4BRAM uses only one port for CIM. This allows the second port to remain available for the DSP to read data simultaneously—enabling a "heterogeneous" compute flow.
  • Duplication Shuffler: A novel circuit that can reorder and replicate weight vectors. This allows the hardware to exploit both weight-sharing and activation-sharing parallelism, which is critical for maintaining high utilization across different neural network topologies.
  • Mixed-Precision Support: It supports 2, 4, or 8-bit weights and variable 2-to-8-bit activations. Because it uses a bit-serial approach for activations, the compute time scales linearly with precision (e.g., 4-bit activations take half the time of 8-bit).

Datasets

The researchers evaluated M4BRAM using the ImageNet classification task across several popular DNN architectures:

  • AlexNet
  • VGG-16
  • ResNet-18 & ResNet-34
  • ViT-Base (Vision Transformer)

They used these models to demonstrate that their quantization schemes maintain high accuracy while reaping significant hardware performance gains.

Key Insights and Ideas

  • True Interoperability: The "aha moment" is that CIM shouldn't be an "either/or" proposition. By keeping one memory port open, M4BRAM acts as a "smart buffer" that computes partial results while feeding the main processor.
  • Linear Performance Scaling: Most FPGA compute resources have "staircase" performance (e.g., you get the same speed for 5-bit as 8-bit). M4BRAM’s bit-serial activation processing ensures that every bit saved in quantization translates directly into a performance boost.
  • Flexibility via Shuffling: Neural networks have diverse shapes. By adding the Duplication Shuffler, the authors allow the BRAM to adapt its internal parallelism ($N_W$ vs $N_I$) to match the specific needs of a layer, preventing hardware "idling."

Key Results

  • Speedup: A heterogeneous accelerator (Hetero-DLA) using M4BRAM achieved an average 2.16× speedup over a baseline accelerator using standard BRAM.
  • Efficiency vs. Prior Art: M4BRAM delivered 1.43× higher performance than BRAMAC (the previous state-of-the-art CIM architecture) because of its superior dataflow and DSP interoperability.
  • Accuracy: The performance gains came with a negligible Top-1 accuracy loss of < 0.5% compared to full-precision floating-point models.
  • Area Overhead: While M4BRAM increases the area of an individual BRAM block by 19-33%, it only increases the total FPGA core area by about 5.6% to 9.5%, a small price for doubling the throughput.

Limitations and Future Work

  • Manual Tiling and Mapping: While the authors developed a design space exploration tool to find optimal tiling, fully automating this into a standard compiler (like TVM or Xilinx Vitis) remains a future challenge.
  • Hardware Specifics: The study is modeled on Intel Stratix-10 (M20K BRAM). While the principles apply to Xilinx FPGAs, the specific area and frequency trade-offs might differ.
  • Workload Partitioning: The current implementation partitions work primarily along the output feature dimension ($Q_{VEC}$). The authors note that exploring partitioning across other dimensions (like input channels) could unlock even more parallelism.

Summary updated

Abstract

Mixed-precision quantization is a popular approach for compressing deep neural networks (DNNs). However, it is challenging to scale the performance efficiently with mixed-precision DNNs given the current FPGA architecture and conventional accelerator dataflows. In this work, we enhance the FPGA’s capability for accelerating mixed-precision DNNs by proposing M4BRAM, a novel compute-in-block RAM (BRAM) architecture that can compute mixed-precision matrix-matrix multiplication. On the precision side, M4BRAM supports a wide range of mixed-precision DNN configurations – the weight precision can be 2/4/8 bits while the activation precision can vary from 2 to 8 bits. On the dataflow side, M4BRAM leverages a novel in-BRAM data duplication scheme to achieve high hardware utilization. Moreover, during M4BRAM computation, other FPGA resources can seamlessly access its data without the need for a separate buffer. Hence, unlike prior compute-in-BRAM proposals, M4BRAM can simultaneously perform mixed-precision computation and maintain full functionality as a memory unit to truly complement the existing compute resources on FPGAs. Experiments show that adding M4BRAM to a tiled DNN accelerator can achieve an average speedup of $2.16\times$ across various DNNs on the ImageNet classification task while incurring a negligible accuracy loss of < 0.5%. Compared to the same tiled accelerator that employs a prior compute-in-BRAM architecture, M4BRAM delivers $1.43\times$ higher performance on average across various DNNs.
BibTeX
@inproceedings{chen2023m,
  author = {Yuzong Chen and Jordan Dotzel and Mohamed S. Abdelfattah},
  title = {{M4BRAM: Mixed-Precision Matrix-Matrix Multiplication in FPGA Block RAMs}},
  booktitle = {ICFPT},
  year = {2023},
  url = {https://openreview.net/forum?id=h1yjHsF08i},
}

Loading related content...

Figure