Introduction
While the acceleration of linear operations like matrix multiplications has seen significant progress through low-precision formats, the "nonlinear" components of Large Language Models (LLMs)—specifically Softmax and Attention—have emerged as critical bottlenecks. As sequence lengths scale, these operations exhibit quadratic growth in computational overhead, often accounting for up to 30% of total inference time. Traditional quantization methods like Block Floating Point (BFP) struggle with these functions due to their high dynamic range; aligning a block to a single maximum value often "crushes" the precision of smaller but numerically significant elements.
The DB-Attn Framework
To address these challenges, researchers introduced DB-Attn, a hardware-software co-design framework centered around a novel data format: Dynamic-BFP (DBFP). Unlike vanilla BFP, which uses a fixed block structure aligned to the maximum exponent, DBFP employs a pivot-focus strategy. By aligning data to a more representative "pivot" (such as the median) and using adaptive grouping, DBFP preserves the precision of elements that are most critical to the nonlinear function's output.
In addition to the data format, the framework utilizes a Dynamic Hierarchical Lookup Table (DH-LUT). This component enables the computation of transcendental functions like $e^x$ using a non-uniform distribution of entries, focusing memory resources on regions where the function changes most rapidly.
Hardware-Software Co-design and Results
The DB-Attn architecture is designed for integer-only computation, bypassing the need for complex and power-hungry floating-point hardware. The implementation on FPGA and ASIC targets demonstrates significant efficiency gains across various modules.
Key performance metrics include:
- Latency Reduction: A 74% reduction in Softmax latency on NVIDIA A800 GPUs.
- Hardware Efficiency: The DBFP divider achieves an 83% reduction in latency compared to traditional FP16 dividers by utilizing shift-addition logic instead of iterative cycles.
- Accuracy Preservation: Testing across LLaMA3-8B showed that DB-Attn maintains an average zero-shot accuracy of 74.26%, nearly identical to the FP16 baseline (74.17%), whereas standard BFP implementations saw accuracy drop to approximately 40%.
Conclusion
DB-Attn demonstrates that BFP-based quantization can be extended to nonlinear LLM operations without sacrificing accuracy. By combining adaptive data formats with specialized hardware units, the framework achieves state-of-the-art efficiency. Future work aims to extend this "BFP-native" approach to other complex layers like GELU, potentially enabling fully quantized end-to-end inference pipelines.
Summary updated