Pushing the Limits of BFP on Narrow Precision LLM Inference

Hui Wang, Yuan Cheng, Xiaomeng Han, Zhengpeng Zhao, Dawei Yang, Zhe Jiang

Introduction

While the acceleration of linear operations like matrix multiplications has seen significant progress through low-precision formats, the "nonlinear" components of Large Language Models (LLMs)—specifically Softmax and Attention—have emerged as critical bottlenecks. As sequence lengths scale, these operations exhibit quadratic growth in computational overhead, often accounting for up to 30% of total inference time. Traditional quantization methods like Block Floating Point (BFP) struggle with these functions due to their high dynamic range; aligning a block to a single maximum value often "crushes" the precision of smaller but numerically significant elements.

The DB-Attn Framework

To address these challenges, researchers introduced DB-Attn, a hardware-software co-design framework centered around a novel data format: Dynamic-BFP (DBFP). Unlike vanilla BFP, which uses a fixed block structure aligned to the maximum exponent, DBFP employs a pivot-focus strategy. By aligning data to a more representative "pivot" (such as the median) and using adaptive grouping, DBFP preserves the precision of elements that are most critical to the nonlinear function's output.

Comparison of standard floating-point formats, Vanilla BFP, and the proposed Dynamic-BFP (DBFP) with adaptive grouping.

In addition to the data format, the framework utilizes a Dynamic Hierarchical Lookup Table (DH-LUT). This component enables the computation of transcendental functions like $e^x$ using a non-uniform distribution of entries, focusing memory resources on regions where the function changes most rapidly.

Hardware-Software Co-design and Results

The DB-Attn architecture is designed for integer-only computation, bypassing the need for complex and power-hungry floating-point hardware. The implementation on FPGA and ASIC targets demonstrates significant efficiency gains across various modules.

The DBFP-driven Softmax hardware architecture and its performance enhancements, showing significant latency reductions in the AdderTree and DIV units.

Key performance metrics include:

  • Latency Reduction: A 74% reduction in Softmax latency on NVIDIA A800 GPUs.
  • Hardware Efficiency: The DBFP divider achieves an 83% reduction in latency compared to traditional FP16 dividers by utilizing shift-addition logic instead of iterative cycles.
  • Accuracy Preservation: Testing across LLaMA3-8B showed that DB-Attn maintains an average zero-shot accuracy of 74.26%, nearly identical to the FP16 baseline (74.17%), whereas standard BFP implementations saw accuracy drop to approximately 40%.

Conclusion

DB-Attn demonstrates that BFP-based quantization can be extended to nonlinear LLM operations without sacrificing accuracy. By combining adaptive data formats with specialized hardware units, the framework achieves state-of-the-art efficiency. Future work aims to extend this "BFP-native" approach to other complex layers like GELU, potentially enabling fully quantized end-to-end inference pipelines.

Summary updated

Abstract

The substantial computational and memory demands of Large Language Models (LLMs) hinder their deployment. Block Floating Point (BFP) has proven effective in accelerating linear operations, a cornerstone of LLM workloads. However, as sequence lengths grow, nonlinear operations, such as Attention, increasingly become performance bottlenecks due to their quadratic computational complexity. These nonlinear operations are predominantly executed using inefficient floating-point formats, which renders the system challenging to optimize software efficiency and hardware overhead. In this paper, we delve into the limitations and potential of applying BFP to nonlinear operations. Given our findings, we introduce a hardware-software co-design framework (DB-Attn), including: (i) DBFP, an advanced BFP version, overcomes nonlinear operation challenges with a pivot-focus strategy for diverse data and an adaptive grouping strategy for flexible exponent sharing. (ii) DH-LUT, a novel lookup table algorithm dedicated to accelerating nonlinear operations with DBFP format. (iii) An RTL-level DBFP-based engine is implemented to support DB-Attn, applicable to FPGA and ASIC. Results show that DB-Attn provides significant performance improvements with negligible accuracy loss, achieving 74% GPU speedup on Softmax of LLaMA and 10x low overhead performance improvement over SOTA designs.
BibTeX
@inproceedings{wang2025pushing,
  author = {Hui Wang and Yuan Cheng and Xiaomeng Han and Zhengpeng Zhao and Dawei Yang and Zhe Jiang},
  title = {{Pushing the Limits of BFP on Narrow Precision LLM Inference}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i20.35407},
}
Figure