Table-Lookup MAC: Scalable Processing of Quantised Neural Networks in FPGA Soft Logic

Daniel Gerlinghoff, Benjamin Chen Ming Choong, Rick Siow Mong Goh, Weng-Fai Wong, Tao Luo

TL;DR

TLMAC (Table-Lookup Multiply-Accumulate) is a novel framework that maps quantized neural network weights directly into FPGA lookup tables (LUTs), effectively turning logic into memory. By combining weight clustering with a hybrid bit-serial architecture, it enables entire ImageNet-scale models to run in FPGA soft logic with 3-bit precision, achieving full-precision accuracy while using significantly fewer resources than previous methods.

What Problem Does This Paper Address?

The "memory wall" is a major bottleneck in AI hardware: moving weights from memory (like BRAM or DDR) to processing units (like DSPs) consumes massive amounts of energy and time. One radical solution is Computing-in-Memory (CIM), where weights are embedded directly into the hardware logic.

On FPGAs, researchers have tried using Lookup Tables (LUTs)—the basic building blocks of FPGA logic—to store weights as part of the logic function itself. However, previous attempts (like LUTNet) faced a "scalability wall." They were mostly limited to Binary Neural Networks (1-bit), which often suffer from poor accuracy. When these older methods tried to scale to larger models or higher bit-widths, the number of LUTs required exploded, and routing the wires between them became an impossible "spaghetti" mess for the FPGA tools to solve.

Methods

The authors introduce TLMAC, which makes LUT-based computing scalable through three key innovations:

  1. Hybrid Bit-Serial Architecture: Instead of trying to compute everything in one massive parallel logic block (which grows exponentially with bit-width), TLMAC processes activations bit-by-bit (serially) while keeping the weights parallel within the LUT. This keeps the "fan-in" (number of inputs) to the LUTs small and manageable.
  2. Weight Group Clustering: Quantized networks have a lot of redundant weight patterns. TLMAC uses spectral clustering to identify these patterns and group them. This allows multiple weights to share the same LUT resources, drastically reducing the total area needed.
  3. Simulated Annealing for Routing: To prevent routing congestion, the authors use a stochastic optimization technique (simulated annealing). This algorithm intelligently places weight groups within the LUT "pool" to minimize the number of physical wires needed to connect the logic to the output accumulators.

Datasets

The researchers evaluated their framework using:

  • ImageNet: The industry-standard, large-scale image classification dataset.
  • ResNet-18: A widely used convolutional neural network architecture.
  • N2UQ (Nonuniform-to-Uniform Quantization): A state-of-the-art quantization method used to prep the models for 2-bit, 3-bit, and 4-bit implementation.

Key Insights and Ideas

  • Redundancy is a Feature, Not a Bug: The authors discovered that in a quantized ResNet-18, unique weight patterns make up less than 5% of the total parameters. TLMAC is designed specifically to exploit this "parameter redundancy" to pack more "intelligence" into the same amount of silicon.
  • Logic as the New Memory: By programming weights into the LUT initializations (the bitstream), the weights never need to be "fetched" during inference. This eliminates weight-transfer power consumption entirely.
  • The Pareto Frontier Shift: Previous soft-logic methods were stuck in the bottom-left of the accuracy-vs-area graph (low accuracy, small area). TLMAC shifts this frontier, achieving high-tier accuracy (70%+) with an area footprint that actually fits on commercially available FPGAs.

Key Results

  • Full Accuracy at 3-Bits: A 3-bit ResNet-18 implemented via TLMAC achieved 71.94% accuracy on ImageNet, matching the accuracy of a full-precision software model.
  • Massive Resource Savings: Compared to previous state-of-the-art LUT-based methods (Logic Shrinkage), TLMAC required 12x fewer LUTs for a 2-bit implementation and 3x fewer for a 4-bit implementation.
  • First of its Kind: This is the first work to successfully fit an entire ImageNet-scale quantized model into the soft logic of a single, off-the-shelf FPGA (the AMD Xilinx Virtex UltraScale+ XCVU13P).

Limitations and Future Work

  • The 4-Bit Ceiling: While 2-bit and 3-bit models fit and routed easily, the 4-bit model hit the physical limits of the FPGA. Even with optimization, the routing congestion in the final layers was too high for standard tools to resolve perfectly.
  • DSP Dependency: While the core MAC operations are in LUTs, the design still relies on DSP slices for "floating-point" tasks like Batch Normalization and skip connections.
  • Future Directions: The authors suggest that more advanced floorplanning (manually telling the FPGA where to put specific blocks) or specialized Place & Route algorithms could help push past the 4-bit congestion limit.

Summary updated

Abstract

Recent advancements in neural network quantisation have yielded remarkable outcomes, with three-bit networks reaching state-of-the-art full-precision accuracy in complex tasks. These achievements present valuable opportunities for accelerating neural networks by computing in reduced precision. Implementing it on FPGAs can take advantage of bit-level reconfigurability, which is not available on conventional CPUs and GPUs. Simultaneously, the high data intensity of neural network processing has inspired computing-in-memory paradigms, including on FPGA platforms. By programming the effects of trained model weights as lookup operations in soft logic, the transfer of weight data from memory units can be avoided, alleviating the memory bottleneck. However, previous methods face poor scalability - the high logic utilisation limiting them to small networks/sub-networks of binary models with low accuracy. In this paper, we introduce Table Lookup Multiply-Accumulate (TLMAC) as a framework to compile and optimise quantised neural networks for scalable lookup-based processing. TLMAC clusters and maps unique groups of weights to lookup-based processing elements, enabling highly parallel computation while taking advantage of parameter redundancy. Further place and route algorithms are proposed to reduce LUT utilisation and routing congestion. We demonstrate that TLMAC significantly improves the scalability of previous related works. Our efficient logic mapping and high degree of reuse enables entire ImageNet-scale quantised models with full-precision accuracy to be implemented using lookup-based computing on one commercially available FPGA.
BibTeX
@inproceedings{gerlinghoff2024table,
  author = {Daniel Gerlinghoff and Benjamin Chen Ming Choong and Rick Siow Mong Goh and Weng-Fai Wong and Tao Luo},
  title = {{Table-Lookup MAC: Scalable Processing of Quantised Neural Networks in FPGA Soft Logic}},
  booktitle = {FPGA},
  year = {2024},
  doi = {10.1145/3626202.3637576},
}

Loading related content...

Figure