BS-Spec: BFP-Based Bit-Sharing for Self-Speculative LLM Decoding with LUT-Driven Mixed-Precision Acceleration

Jiwoo Kim, Gunho Park, Sunwoo Yoo, Baeseong Park, Dongsoo Lee, Youngjoo LeearXiv 2026 · arXiv

TL;DR

BS-Spec is a software-hardware co-design that accelerates Large Language Model (LLM) inference using "bit-sharing" self-speculative decoding. By representing models in a Block Floating Point (BFP) format, it derives multiple low-precision draft models from a single high-precision target model on the fly, dynamically adjusting precision to match the current workload and achieving up to a 1.9× speedup with significantly lower energy consumption.

What Problem Does This Paper Address?

LLM inference is notoriously memory-bound. While speculative decoding—using a tiny "draft" model to guess tokens that a large "target" model then verifies—can speed things up, it introduces new headaches. Conventional speculative decoding requires storing a separate draft model, eating up precious GPU memory.

Existing "self-speculative" methods try to avoid this by using the target model itself as a drafter (e.g., by skipping layers or using low-bit quantization). However, these methods often use a fixed configuration that fails to adapt to diverse serving scenarios. For instance, a configuration optimized for short sequences might be terribly inefficient for long-context windows where the KV cache dominates memory traffic. The authors identify a need for a system that can jointly optimize draft construction, workload-aware precision, and hardware execution.

Methods

The researchers propose a three-pronged approach:

  1. BFP-Based Bit-Sharing: Instead of standard floating-point formats, BS-Spec uses Block Floating Point (BFP). In BFP, a group of numbers shares a single exponent, and their significands are aligned. This allows the system to create a draft model simply by loading the Most Significant Bits (MSBs) of the target model's weights and KV cache. No separate storage or complex conversion is needed.

    ![Fig.

  2. Procedure to generate BFP format with per-group shared exponent.](/figures/art_5ae4c9b133b549beb7006b1a9a2e6d7a)

  3. Workload-Aware Precision Selection: The optimal number of bits to load for the draft model changes based on the batch size ($B$) and sequence length ($S$). BS-Spec uses an offline calibration phase to find the "Pareto-optimal" precision pairs for weights ($p_w$) and KV cache ($p_{kv}$) across different workloads. These are stored in a Precision Lookup Table (PLT) that the controller queries at runtime.

    ![Fig.

  4. Overview of BS-Spec software-hardware co-optimization flow.](/figures/art_4ae0be6b909d4579aded689146ac635a)

  5. Bit-Serial Accelerator: To actually benefit from variable bit-widths, the authors designed a custom hardware accelerator. Unlike standard bit-parallel hardware, this bit-serial design processes data one bit-plane at a time. If you only load 5 bits for a draft, the hardware finishes in roughly 5 cycles; if you load 9 bits for the target, it takes 9. It uses LUT-based accumulation (BS-RAC) to replace power-hungry multipliers, folding the BFP exponent alignment directly into the lookup table construction.

Datasets

The authors evaluated BS-Spec using three prominent LLMs:

  • Llama-3.2-3B
  • Llama-3.1-8B
  • Qwen3-14B

To measure the "acceptance rate" (how often the draft model's guesses were correct) and overall accuracy, they used standard benchmarks:

  • HumanEval (Coding tasks)
  • GSM8K (Grade school math word problems)
  • MBPP (Mostly Basic Python Problems)

Key Insights and Ideas

  • The "Moving Bottleneck": A crucial insight is that the memory bottleneck shifts. In short-context regimes, loading model weights is the primary cost. In long-context regimes, the KV cache becomes the elephant in the room. BS-Spec is the first to jointly adapt both weight and KV cache precision to handle this shift.
  • BFP as the Enabler: By using BFP, the draft model is literally a "subset" of the target model. This eliminates the need for separate draft model weights, which the authors show can be a major limitation for other state-of-the-art methods like QuantSpec.
  • Hardware-Software Synergy: Bit-serial hardware is often dismissed as slow, but in a memory-bound speculative decoding context, its ability to perfectly scale execution time with bit-width makes it a perfect match for variable-precision drafting.

Key Results

  • Speedup: BS-Spec achieved up to 1.9× speedup over standard autoregressive decoding.
  • Efficiency: It improved speedup over existing bit-sharing baselines (like SPEQ and QuantSpec) by 67.2% on average.
  • Energy: The system reduced energy consumption per token to as low as 0.43× of the baseline.
  • Accuracy Preservation: Even when using a compressed BFP9 target model, the accuracy on benchmarks like GSM8K remained nearly identical to the original BF16 model (e.g., 77.8% vs 77.7% for Llama-3.2-3B).

Limitations and Future Work

  • KV Cache Buffering: Because BFP requires grouping values to share an exponent, KV cache entries must be temporarily buffered until a full group is formed before conversion. While the authors state this overhead is small, it adds a layer of complexity to the memory controller.
  • Hardware Specificity: The gains are most pronounced on the custom BS-Spec accelerator. While the bit-sharing logic could be applied to GPUs, standard GPU cores aren't optimized for bit-serial execution, likely limiting the potential speedup on commodity hardware.
  • Future Directions: The authors suggest exploring even more aggressive bit-sharing schemes or applying these techniques to other model architectures like State Space Models (SSMs).

For more details, you can find the full paper on arXiv under the identifier 2610.22877v1.

Summary updated

Abstract

As large language models (LLMs) are increasingly deployed across diverse inference services, serving workloads span a wide range of batch sizes, context lengths, and performance requirements. Speculative decoding improves inference throughput, but conventional approaches require a separate draft model and KV cache, while existing self-speculative methods suffer from workload-insensitive draft configurations and inefficient mixed-precision execution. In this paper, we present BS-Spec, a block floating point (BFP)-based bit-sharing framework that jointly optimizes draft construction, workload-aware precision selection, and hardware execution. BS-Spec derives multiple draft representations directly from a single BFP target model by partially loading the most significant bits of weights and the KV cache while preserving shared exponents, eliminating separate draft-model storage. Offline calibration selects the weight and KV-cache precision pair that maximizes estimated speedup for each serving regime, and a precision lookup table applies the selected configuration at runtime. To support this adaptive execution, we design a mixed-precision BFP accelerator with a lookup table (LUT)-based variable-precision datapath for both draft and target computation. Experimental results show that BS-Spec achieves up to $1.9\times$ speedup over conventional autoregressive decoding and improves speedup over the evaluated bit-sharing self-speculative decoding baselines by $67.2\%$ on average, demonstrating the benefits of jointly co-designing bit sharing, precision adaptation, and specialized hardware for diverse LLM serving workloads.
BibTeX
@misc{kim2026bs,
  author = {Jiwoo Kim and Gunho Park and Sunwoo Yoo and Baeseong Park and Dongsoo Lee and Youngjoo Lee},
  title = {{BS-Spec: BFP-Based Bit-Sharing for Self-Speculative LLM Decoding with LUT-Driven Mixed-Precision Acceleration}},
  year = {2026},
  eprint = {2610.22877},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.22877},
}
Figure