Burstiness, Not Bit Rate: Making Adaptive Weight Reconstruction Predictable on Real Hardware

Large language models run slowly because the chip spends most of its time waiting for weights to arrive from off-chip memory. One response is on-the-fly weight reconstruction: store each block of weights as a tiny recipe, such as a random-number seed plus a few coefficients or an entropy-coded bitstream, and rebuild the full block with cheap logic right next to the multipliers just before it is used. The strongest recent schemes spend their bit budget unevenly. They give "hard" blocks more basis vectors or longer codes and "easy" blocks fewer. This lowers the average bits per weight. It also means that the time to rebuild a block varies from block to block. The architectural simulators used to evaluate these designs typically model reconstruction as a steady stream at the average rate, so the variation is invisible to them.

We propose that, on real hardware, throughput under adaptive reconstruction depends on the worst run of expensive blocks that the on-chip buffer must absorb, not on the average bit rate. We will measure how far simulator predictions drift from measured FPGA throughput as a function of this burstiness. We will then change the compressor so that it allocates bits under an explicit burstiness budget matched to the buffer the hardware actually has. If this works, it gives the stall-free throughput that the simulators promise, at close to the accuracy of unconstrained adaptive allocation, and a calibrated performance model that future reconstruction papers can check their speedup claims against.

Core Claim

Consider a weight generator feeding a systolic array, which is a grid of multiply-accumulate units that passes data between neighbors in lockstep. Write $c_i$ for the cycles needed to rebuild block $i$ in streaming order, $C$ for the cycles the array spends consuming one block, and $B$ for the slack, in cycles, held in the prefilled buffer between them. The array never stalls if and only if

$$\max_{j \le l} \sum_{i=j}^{l} \big(c_i - C\big) \le B,$$

that is, the reconstruction deficit accumulated over any run of consecutive blocks never exceeds the buffer slack. This is the standard leaky-bucket condition from queueing and network calculus. Average-rate models check only whether the mean of $c_i$ is at most $C$, and that mean condition is necessary but not sufficient.

Our claim has two parts. First, adaptive allocation of the kind used in S-Quant concentrates high-cost blocks spatially, because high-variance blocks cluster in particular layers, heads, and channels. Existing designs therefore can violate this condition even when their average rate fits. In a synchronous array, a starved generator stalls the whole wavefront rather than a single processing element. Second, adding the condition as a constraint inside the bit-allocation step recovers most of the accuracy benefit of adaptivity while keeping measured throughput close to the ideal. What would be new is treating reconstruction-cost burstiness as a first-class compression objective, and validating that choice on measured hardware rather than an idealized cycle model.

Grounding in Prior Work

The reconstruction literature supplies the method and the motivation.

Three papers supply the mechanism and the hardware evidence we build on. SeedLM approximates each weight block as pseudo-random basis columns from a shift-register generator times learned coefficients. It uses a fixed number of coefficients per block and reports 3.67× to 3.98× measured FPGA speedup over FP16. Its uniform rate is exactly why it has no burstiness problem, and it serves as our uniform-rate reference. S-Quant adds adaptive basis allocation driven by an explained-energy ratio. It reports a 4× speedup and about 3% generator area, but only from a custom-accelerator simulation, and it is the closest prior work. We keep its allocation idea and ask what happens when the stream becomes bursty on real hardware. unzipFPGA builds on-chip weight generation from orthogonal codes on Zynq devices. Its analytical design-space model is calibrated to FPGA resources, which makes it the right template for our calibrated model.

Two GPU results supply evidence that variable-length decoding already costs real performance. EntQuant entropy-codes 8-bit weights and decodes them on the fly. It runs roughly 1.5–2× slower than BFloat16 despite moving far fewer bytes. ECF8 Huffman-codes floating-point exponents and reports throughput gains anywhere from 11.3% to 177.1% across models. That wide spread is consistent with decode cost that is sensitive to workload structure rather than to bit rate alone.

The simulation-gap literature constrains how we validate. SCALE-Sim models systolic arrays with idealized double-buffered SRAM and is the simulator whose predictions we will test. DRACO shows that theoretical throughput on systolic arrays diverges sharply from achieved throughput because of idle processing elements. We ask the analogous question for reconstruction-induced idling. Walker et al. show that event-based simulator power models fail outside their calibration regime, and that correcting them against real counters brings error to 2.8% MAPE. We borrow their methodology of calibrating a model against hardware counters rather than trusting it.

We deliberately do not claim to measure custom-chip effects such as IR-drop or voltage droop. The file notes that FPGA logic does not reflect those dynamics. Overclaiming them without silicon would repeat the very gap we study.

Research Program

The program has three parts.

The first part is measurement. We will implement an FPGA accelerator with a shift-register weight generator feeding a weight-stationary systolic array through a FIFO of configurable depth, and instrument it with cycle counters for generator-busy, array-stall, and buffer occupancy. Onto it we will load SeedLM-style uniform models and S-Quant-style adaptive models at matched average bit rates of 3 and 4 bits. For each configuration we will record the measured cycle counts, the SCALE-Sim prediction, and the burstiness statistic on the left side of the condition above. The expected finding is that simulator error grows with burstiness relative to $B$ and is near zero for uniform allocation. If that relationship does not appear, the rest of the program is unnecessary, and we will report it as a negative result.

The second part is the constrained compressor. We modify the per-block basis-count search so that it maximizes total explained energy subject to the leaky-bucket condition at the hardware's actual $B$ and streaming order. Streaming order is taken from the dataflow, so the constraint is the true one, not a proxy. Because the condition is a prefix-sum constraint along a sequence, it can be solved exactly with dynamic programming over (block, current deficit) states, discretized at cycle granularity. For entropy-coded weights the same constraint caps the code length of each decode chunk. Chunks that exceed the cap fall back to a longer fixed-length encoding, which trades a little compression for bounded decode time. The compressor also gets a limited freedom to reorder tiles where the dataflow permits it, and this reordering is ablated separately.

The third part is a calibrated model. We extend a SCALE-Sim-style model with a generator-queue stage whose per-block service times are read from the compressed model. We fit it to the FPGA counters and test it on held-out models and array sizes. This gives other authors a cheap check that does not require hardware.

Baselines

The naive reference point is uncompressed FP16 inference on the same array, alongside a SeedLM-style uniform-rate model at the same average bit rate. The uniform model has no burstiness by construction. It represents what one gets by simply refusing to adapt, and its reported figures are 3.67× to 3.98× FPGA speedup over FP16 and 97.9% zero-shot accuracy retention at 4 bits on Llama 3 70B (SeedLM). We will reproduce it on our own board rather than compare against numbers from a different FPGA.

The strongest existing method is S-Quant's unconstrained adaptive allocation. Its reported number is a 4× simulated speedup for memory-bound LLM inference (S-Quant). We found no reported measured-hardware result for it. It is the strongest competitor because it has the best accuracy per bit among the seed-based schemes in the file. We will run it with the same seed search, block size, average bit budget, and search compute as our constrained version. The only difference will be the burstiness constraint.

A resource-matched variant gives S-Quant a FIFO deep enough to absorb its measured worst-case deficit. The extra buffer memory is charged against the design by reducing array size or clock to keep FPGA resource use equal.

For accuracy, we compare against native INT4 round-to-nearest quantization on the same model, which has no reconstruction overhead at all. Where our budget allows, we also include EntQuant at an effective 3 bits. EntQuant is reported to be within about 1.6% of the uncompressed LLaMA-2 70B on LM Eval (EntQuant). No reported WikiText-2 perplexity values for these methods were found on Lacuna, so all perplexities will be measured in-house.

The key ablation separates constraint from reordering. It runs three versions: order-only (tile reordering, unconstrained allocation), constraint-only (fixed order), and both.

Our method counts as beating the baselines only if it meets two conditions together. It must reach measured throughput within 5% of the stall-free ideal at a FIFO depth where unconstrained S-Quant stalls. It must also keep WikiText-2 perplexity and LM Eval accuracy closer to unconstrained S-Quant than to the uniform-rate model.

Why not just…?

Why not just use a deeper FIFO? A deep enough buffer always satisfies the condition, so this is the most direct threat. The cost is on-chip memory, and on-chip memory is what these designs exist to save. SCALE-Sim also reports diminishing returns from larger buffers. The resource-matched baseline settles the question. If the depth needed to absorb S-Quant's bursts costs less than about 1% of array throughput after reallocating resources, the constraint is unnecessary, and we will say so.

Why not just use uniform allocation, as SeedLM does? Uniform allocation is burst-free but gives up the accuracy that motivated adaptivity. The relevant question is how large that accuracy gap is at 3 bits, where S-Quant claims adaptivity matters most. If uniform and adaptive models differ by less than measurement noise on WikiText-2 and LM Eval at matched rate, the whole problem is moot. We therefore measure this gap first.

Why not just reorder the tiles to interleave heavy and light blocks? This is a real threat. It is a scheduling pass that leaves the compressor untouched, and it may flatten most bursts on its own. The order-only ablation tests it directly. If reordering plus a small FIFO matches the full method, our contribution shrinks to the measured simulator gap, the calibrated model, and a scheduling recommendation. That outcome is still useful, but it is smaller. Reordering is limited, though, because the dataflow fixes much of the streaming order, and heavy blocks concentrate within individual layers that must be processed contiguously.

Why not just fix the simulator? We do, in the third part of the program. But a corrected simulator only reports stalls. It does not remove them, and the compressor makes its allocation decisions without ever seeing the simulator. The contribution is to feed the hardware condition back into allocation.

Decisive Checks

The first check comes from the measurement phase. Across matched-rate uniform and adaptive models, SCALE-Sim's throughput error must correlate with the burstiness-to-buffer ratio and must be near zero for uniform models. Otherwise the premise fails.

The second check is the paired comparison at fixed average rate and fixed FIFO depth. Constrained allocation must close most of the measured stall gap. Its perplexity cost relative to unconstrained adaptive allocation must be smaller than the cost of falling back to uniform allocation.

The third check tests the calibrated model. Its throughput predictions on held-out models and array sizes should be within a few percent of the counters, in the spirit of the 2.8% error reported by Walker et al.. A model that cannot reach this on held-out configurations is not a usable proxy.

The fourth check is a secondary portability probe. We apply the chunk-length cap to entropy-coded weights on a GPU to see whether it narrows EntQuant's reported 1.5–2× slowdown. This would indicate whether burstiness explains part of that overhead on commodity hardware as well.

Workloads are LLaMA-family models evaluated on WikiText-2 and the LM Evaluation Harness tasks (ARC, HellaSwag, WinoGrande, BoolQ), the benchmarks shared by SeedLM, S-Quant, and EntQuant.