M4BRAM: Mixed-Precision Matrix-Matrix Multiplication in FPGA Block RAMs

Yuzong Chen, Jordan Dotzel, Mohamed S. Abdelfattah

Abstract

Mixed-precision quantization is a popular approach for compressing deep neural networks (DNNs). However, it is challenging to scale the performance efficiently with mixed-precision DNNs given the current FPGA architecture and conventional accelerator dataflows. In this work, we enhance the FPGA's capability for accelerating mixed-precision DNNs by proposing M4BRAM, a novel compute-in-block RAM (BRAM) architecture that can compute mixed-precision matrix-matrix multiplication. On the precision side, M4BRAM supports a wide range of mixed-precision DNN configurations—the weight precision can be 2/4/8 bits while the activation precision can vary from 2 to 8 bits. On the dataflow side, M4BRAM leverages a novel in-BRAM data duplication scheme to achieve high hardware utilization. Moreover, during M4BRAM computation, other FPGA resources can seamlessly access its data without the need for a separate buffer. Hence, unlike prior compute-in-BRAM proposals, M4BRAM can simultaneously perform mixed-precision computation and maintain full functionality as a memory unit to truly complement the existing compute resources on FPGAs. Experiments show that adding M4BRAM to a tiled DNN accelerator can achieve an average speedup of 2.16× across various DNNs on the ImageNet classification task while incurring a negligible accuracy loss of < 0.5%. Compared to the same tiled accelerator that employs a prior compute-in-BRAM architecture, M4BRAM delivers 1.43× higher performance on average across various DNNs.

Introduction

Deep Neural Networks (DNNs) are increasingly utilizing mixed-precision quantization—where different layers or operands (weights vs. activations) use varying bit-widths—to optimize power consumption and memory footprint. Despite these algorithmic advances, current FPGA architectures struggle to translate these savings into proportional performance gains. Standard Digital Signal Processing (DSP) blocks are typically optimized for fixed, high-precision arithmetic. "Packing" multiple low-precision operations into a single DSP is often inefficient and fails to scale linearly as precision decreases.

Previous attempts to address this via Compute-In-Memory (CIM) in Block RAMs (BRAMs) introduced a significant trade-off: they typically "locked" the BRAM during computation, preventing it from serving as a standard memory buffer. This forced designers to choose between using BRAM for compute or using it to feed the DSPs, limiting the system's overall throughput. M4BRAM is designed to overcome these limitations by transforming BRAMs into flexible, mixed-precision engines that complement existing DSP resources.

The M4BRAM Architecture

M4BRAM introduces a modified BRAM architecture that maintains its role as a storage element while adding high-performance compute capabilities. The design is centered around several key technical innovations:

Decoupled Processing Elements (BPEs)

Instead of modifying the internal memory array, M4BRAM adds four small BRAM Processing Elements (BPEs) to the BRAM periphery. These units handle the arithmetic operations, ensuring the main memory array remains largely undisturbed and accessible.

Single-Port CIM and Heterogeneous Flow

Unlike prior CIM architectures that occupied both BRAM ports for computation, M4BRAM utilizes only one port for its internal matrix multiplication. This leaves the second port available for the rest of the FPGA fabric. This allows for a "heterogeneous" compute flow where the BRAM acts as a "smart buffer," computing partial results internally while simultaneously feeding data to DSP blocks for other operations.

Duplication Shuffler and Flexibility

Neural network layers vary significantly in their dimensions. M4BRAM incorporates a novel "Duplication Shuffler" circuit that can reorder and replicate weight vectors. This allows the hardware to exploit both weight-sharing and activation-sharing parallelism, ensuring high hardware utilization regardless of the specific network topology.

Mixed-Precision Support

M4BRAM is designed for the diverse landscape of modern quantization. It supports weight precisions of 2, 4, or 8 bits and activation precisions ranging from 2 to 8 bits. By employing a bit-serial approach for activations, M4BRAM achieves linear performance scaling. For example, a 4-bit activation layer will compute twice as fast as an 8-bit layer, a feat that standard FPGA DSPs—which often have "staircase" performance characteristics—cannot easily replicate.

Experimental Evaluation

The researchers evaluated M4BRAM using the ImageNet classification task across a variety of architectures, including AlexNet, VGG-16, ResNet-18, ResNet-34, and the Vision Transformer (ViT-Base). The study focused on three primary metrics: performance speedup, accuracy preservation, and hardware overhead.

Performance and Efficiency

A heterogeneous accelerator (Hetero-DLA) integrated with M4BRAM achieved an average 2.16× speedup over a baseline accelerator using standard BRAM. When compared to BRAMAC, the previous state-of-the-art CIM architecture, M4BRAM delivered 1.43× higher performance. This efficiency is largely attributed to M4BRAM's ability to operate in parallel with DSPs rather than competing for memory access.

Accuracy and Area

The performance gains were achieved with minimal impact on model quality, showing a Top-1 accuracy loss of less than 0.5% compared to full-precision floating-point models. In terms of hardware cost, while M4BRAM increases the area of an individual BRAM block by 19–33%, the total FPGA core area increase is estimated at only 5.6% to 9.5%.

Conclusion

M4BRAM represents a significant step forward in FPGA-based DNN acceleration. By enabling BRAMs to perform mixed-precision matrix multiplication without sacrificing their functionality as memory buffers, the architecture allows for a truly heterogeneous compute environment. This approach effectively doubles throughput for modern, quantized neural networks with negligible area overhead and accuracy loss, providing a scalable path for future mixed-precision AI hardware.

Summary updated

Abstract

Mixed-precision quantization is a popular approach for compressing deep neural networks (DNNs). However, it is challenging to scale the performance efficiently with mixed-precision DNNs given the current FPGA architecture and conventional accelerator dataflows. In this work, we enhance the FPGA's capability for accelerating mixed-precision DNNs by proposing M4BRAM, a novel compute-in-block RAM (BRAM) architecture that can compute mixed-precision matrix-matrix multiplication. On the precision side, M4BRAM supports a wide range of mixed-precision DNN configurations -- the weight precision can be 2/4/8 bits while the activation precision can vary from 2 to 8 bits. On the dataflow side, M4BRAM leverages a novel in-BRAM data duplication scheme to achieve high hardware utilization. Moreover, during M4BRAM computation, other FPGA resources can seamlessly access its data without the need for a separate buffer. Hence, unlike prior compute-in-BRAM proposals, M4BRAM can simultaneously perform mixed-precision computation and maintain full functionality as a memory unit to \textit{truly} complement the existing compute resources on FPGAs. Experiments show that adding M4BRAM to a tiled DNN accelerator can achieve an average speedup of 2.16$\times$ across various DNNs on the ImageNet classification task while incurring a negligible accuracy loss of $<$ 0.5%. Compared to the same tiled accelerator that employs a prior compute-in-BRAM architecture, M4BRAM delivers 1.43$\times$ higher performance on average across various DNNs.
BibTeX
@inproceedings{chen2023m,
  author = {Yuzong Chen and Jordan Dotzel and Mohamed S. Abdelfattah},
  title = {{M4BRAM: Mixed-Precision Matrix-Matrix Multiplication in FPGA Block RAMs}},
  booktitle = {ICFPT},
  year = {2023},
  url = {https://openreview.net/forum?id=h1yjHsF08i},
}
Figure