Introduction
The deployment of massive Large Language Models (LLMs) on resource-constrained edge devices, such as Field-Programmable Gate Arrays (FPGAs), is fundamentally limited by memory bandwidth and computational capacity. While Post-Training Quantization (PTQ) is a standard technique for compressing 32-bit floating-point weights into lower-precision formats, pushing precision below 8 bits (e.g., 4-bit weights and 8-bit activations) typically results in a significant accuracy drop—often exceeding 5% in BLEU scores for translation tasks. ITERA-LLM addresses this by introducing a software-hardware co-design framework that combines iterative tensor decomposition with hardware-aware optimization.
Methodology
The ITERA-LLM framework bridges the gap between algorithmic compression and hardware execution through three primary components: iterative SVD, sensitivity-based rank allocation, and a co-design search space.
Iterative SVD-Based Tensor Decomposition
Standard Singular Value Decomposition (SVD) approximates weight matrices but often fails to account for quantization noise. ITERA-LLM employs an iterative approach where singular vectors are extracted and quantized, and the resulting residual error is used as the input for the subsequent SVD step. This closed-loop process allows the model to actively compensate for errors introduced during the quantization of previous ranks.
Sensitivity-Based Rank Allocation (SRA)
Not all layers within a transformer architecture contribute equally to model performance. ITERA-LLM utilizes a finite difference method to estimate the sensitivity of global accuracy to compression at each layer. By dynamically allocating higher SVD ranks to sensitive layers and lower ranks to more robust ones, the framework optimizes accuracy within a fixed memory budget.
Hardware-Software Co-Design
To ensure algorithmic gains translate to real-world performance, ITERA-LLM incorporates analytical models for FPGA resource utilization (DSPs, BRAMs) and latency. A Design Space Exploration (DSE) algorithm optimizes both algorithmic parameters (ranks, bit-widths) and hardware parameters (tiling, parallelization). Crucially, the framework favors constant bit-widths with variable ranks, which maximizes FPGA DSP block utilization compared to mixed-precision schemes that require complex bit-serial hardware.
Experimental Results
The framework was evaluated using OPUS-MT models on the WMT2019 dataset for English-to-German (EN-DE) and French-to-English (FR-EN) translation tasks.
- Accuracy Improvements: At extreme W4A8 precision, ITERA-LLM achieved up to a 4.9% BLEU score improvement over standard quantization-only baselines at equivalent compression ratios.
- Computational Efficiency: For a given accuracy target at W6A8 precision, the SRA method reduced the total number of fixed-point operations by 12.5%.
- Hardware Speedup: Deployment on a Xilinx ZCU111 FPGA demonstrated latency reductions of 12.1% to 41.1% for linear layers compared to optimized quantization-only MatMul engines.
Conclusion and Future Work
ITERA-LLM demonstrates that iterative tensor decomposition, when coupled with hardware-aware rank allocation, can effectively mitigate the accuracy loss associated with sub-8-bit quantization. This approach enables more efficient LLM deployment on edge devices without the overhead of Quantization-Aware Training. Future research will focus on extending this framework to larger generative models (e.g., LLaMA) and multi-FPGA systems.
Summary updated