Abstract
Conventional multiply-accumulate (MAC) operations have long dominated computation time for deep neural networks (DNNs). Recently, product quantization (PQ) has been successfully applied to these workloads, replacing MACs with memory lookups to pre-computed dot products. While this property makes PQ an attractive solution for model acceleration, little is understood about the associated trade-offs in terms of compute and memory footprint, and the impact on accuracy. Our empirical study investigates the impact of different PQ settings and training methods on layerwise reconstruction error and end-to-end model accuracy. When studying the efficiency of deploying PQ DNNs, we find that metrics such as FLOPs, number of parameters, and even CPU/GPU performance, can be misleading. To address this issue, and to more fairly assess PQ in terms of hardware efficiency, we design the first custom hardware accelerator to evaluate the speed and efficiency of running PQ models. We identify PQ configurations that are able to improve performance-per-area for ResNet20 by 40%-104%, even when compared to a highly optimized conventional DNN accelerator. Our hardware performance outperforms recent PQ solutions by 4x, with only a 0.6% accuracy degradation. This work demonstrates the practical and hardware-aware design of PQ models, paving the way for wider adoption of this emerging DNN approximation methodology.
Introduction
Deep Neural Networks (DNNs) are notoriously compute-heavy, typically requiring billions of multiply-accumulate (MAC) operations. While standard linear quantization (e.g., 8-bit integer) reduces precision, it does not change the fundamental computational nature of the workload. Product Quantization (PQ) offers a radical alternative by approximating inputs through a set of learned prototypes. By pre-computing dot products between these prototypes and network weights, PQ replaces power-hungry multiplications with efficient memory lookups (LUTs).
However, standard hardware like CPUs and GPUs are optimized for arithmetic throughput, not massive, parallel memory lookups. On these platforms, PQ-DNNs often run slower than traditional models, creating a "hardware-software gap." This research addresses this gap by designing a custom hardware accelerator specifically for PQ.
Product Quantization Fundamentals
PQ works by splitting input vectors into several subspaces. For each subspace, the input is matched to the nearest "prototype" from a codebook. The dot product for the entire operation is then reconstructed by summing the pre-computed partial results stored in a Look-Up Table (LUT).
As shown in the figure above, the inference process involves finding the index of the closest prototype and using it to index into a pre-computed LUT. This shift from compute-bound to memory-bound operations necessitates a rethinking of hardware architecture.
The Product Quantization Accelerator (PQA)
To unlock the potential of PQ, the researchers developed the Product Quantization Accelerator (PQA), implemented on Intel Agilex FPGAs. The architecture is composed of three specialized modules designed to handle the unique memory patterns of PQ models.
- Distance Calculation: This module performs a nearest-neighbor search (using L1 or L2 distance) to identify which prototype best represents the input segment. The researchers found that this search can be performed at very low precision (2–6 bits) without significant accuracy loss.
- Product Lookup: This is a highly partitioned memory array that enables massive parallelism, fetching partial results for multiple output channels simultaneously.
- Accumulator: This module sums the partial results from different subspaces to produce the final output.
Empirical Evaluation and Insights
The team evaluated PQA across several tasks: image classification (ResNet20 on CIFAR-10), character recognition (DW-CNN on EMNIST), and keyword spotting (MicroNet on SpeechCommands).
A key finding is that FLOPs are a deceptive metric for PQ. Because PQ replaces computation with memory access, a model with fewer FLOPs might be slower on standard hardware if the memory subsystem is not optimized for lookups. Custom memory banking is mandatory; by partitioning the LUT across many small on-chip memories, PQA can perform hundreds of lookups in a single clock cycle—a feat standard CPU caches cannot match.
Key Results
- Efficiency Gains: PQA achieved up to a 3.1× improvement in performance-per-area compared to a highly optimized conventional Deep Learning Accelerator (DLA).
- State-of-the-Art Comparison: PQA delivered 4× better performance-per-area than the recent PECAN PQ solution, with only a negligible 0.6% drop in accuracy.
- Accuracy Preservation: On keyword spotting tasks, the team achieved 95% accuracy (within 0.3% of the baseline) while reducing the computational footprint by 2.4×.
Conclusion and Future Work
While PQA demonstrates significant hardware advantages, training PQ-DNNs remains a challenge. The process is computationally intensive and sensitive to hyperparameters like the "temperature" used during encoding transitions. Future research will focus on making PQ training more robust and exploring combinations with other compression techniques, such as channel pruning, to further optimize the lookup tables for even larger models.
Summary updated