TL;DR
BS-Spec is a software-hardware co-design that accelerates Large Language Model (LLM) inference using "bit-sharing" self-speculative decoding. By representing models in a Block Floating Point (BFP) format, it derives multiple low-precision draft models from a single high-precision target model on the fly, dynamically adjusting precision to match the current workload and achieving up to a 1.9× speedup with significantly lower energy consumption.
What Problem Does This Paper Address?
LLM inference is notoriously memory-bound. While speculative decoding—using a tiny "draft" model to guess tokens that a large "target" model then verifies—can speed things up, it introduces new headaches. Conventional speculative decoding requires storing a separate draft model, eating up precious GPU memory.
Existing "self-speculative" methods try to avoid this by using the target model itself as a drafter (e.g., by skipping layers or using low-bit quantization). However, these methods often use a fixed configuration that fails to adapt to diverse serving scenarios. For instance, a configuration optimized for short sequences might be terribly inefficient for long-context windows where the KV cache dominates memory traffic. The authors identify a need for a system that can jointly optimize draft construction, workload-aware precision, and hardware execution.
Methods
The researchers propose a three-pronged approach:
-
BFP-Based Bit-Sharing: Instead of standard floating-point formats, BS-Spec uses Block Floating Point (BFP). In BFP, a group of numbers shares a single exponent, and their significands are aligned. This allows the system to create a draft model simply by loading the Most Significant Bits (MSBs) of the target model's weights and KV cache. No separate storage or complex conversion is needed.

-
Workload-Aware Precision Selection: The optimal number of bits to load for the draft model changes based on the batch size ($B$) and sequence length ($S$). BS-Spec uses an offline calibration phase to find the "Pareto-optimal" precision pairs for weights ($p_w$) and KV cache ($p_{kv}$) across different workloads. These are stored in a Precision Lookup Table (PLT) that the controller queries at runtime.

- Bit-Serial Accelerator: To actually benefit from variable bit-widths, the authors designed a custom hardware accelerator. Unlike standard bit-parallel hardware, this bit-serial design processes data one bit-plane at a time. If you only load 5 bits for a draft, the hardware finishes in roughly 5 cycles; if you load 9 bits for the target, it takes 9. It uses LUT-based accumulation (BS-RAC) to replace power-hungry multipliers, folding the BFP exponent alignment directly into the lookup table construction.
Datasets
The authors evaluated BS-Spec using three prominent LLMs:
- Llama-3.2-3B
- Llama-3.1-8B
- Qwen3-14B
To measure the "acceptance rate" (how often the draft model's guesses were correct) and overall accuracy, they used standard benchmarks:
- HumanEval (Coding tasks)
- GSM8K (Grade school math word problems)
- MBPP (Mostly Basic Python Problems)
Key Insights and Ideas
- The "Moving Bottleneck": A crucial insight is that the memory bottleneck shifts. In short-context regimes, loading model weights is the primary cost. In long-context regimes, the KV cache becomes the elephant in the room. BS-Spec is the first to jointly adapt both weight and KV cache precision to handle this shift.
- BFP as the Enabler: By using BFP, the draft model is literally a "subset" of the target model. This eliminates the need for separate draft model weights, which the authors show can be a major limitation for other state-of-the-art methods like QuantSpec.
- Hardware-Software Synergy: Bit-serial hardware is often dismissed as slow, but in a memory-bound speculative decoding context, its ability to perfectly scale execution time with bit-width makes it a perfect match for variable-precision drafting.
Key Results
- Speedup: BS-Spec achieved up to 1.9× speedup over standard autoregressive decoding.
- Efficiency: It improved speedup over existing bit-sharing baselines (like SPEQ and QuantSpec) by 67.2% on average.
- Energy: The system reduced energy consumption per token to as low as 0.43× of the baseline.
- Accuracy Preservation: Even when using a compressed BFP9 target model, the accuracy on benchmarks like GSM8K remained nearly identical to the original BF16 model (e.g., 77.8% vs 77.7% for Llama-3.2-3B).
Limitations and Future Work
- KV Cache Buffering: Because BFP requires grouping values to share an exponent, KV cache entries must be temporarily buffered until a full group is formed before conversion. While the authors state this overhead is small, it adds a layer of complexity to the memory controller.
- Hardware Specificity: The gains are most pronounced on the custom BS-Spec accelerator. While the bit-sharing logic could be applied to GPUs, standard GPU cores aren't optimized for bit-serial execution, likely limiting the potential speedup on commodity hardware.
- Future Directions: The authors suggest exploring even more aggressive bit-sharing schemes or applying these techniques to other model architectures like State Space Models (SSMs).
For more details, you can find the full paper on arXiv under the identifier 2610.22877v1.
Summary updated