Scientific instruments now produce data faster than any system can store it. The Square Kilometre Array is expected to output about 800 GB/s, and collider triggers must decide within a hundred nanoseconds what to keep. The obvious fix is to spend fewer bits per number. Below about 3 bits per value, though, quantization collapses: a tiny grid cannot hold both rare large values and the bulk of small ones. The best-known fix from language-model compression rotates the data with a random orthogonal matrix so that outlier energy spreads evenly across all coordinates. We argue that this fix is wrong for most scientific signals, because their large values are the science. A calorimeter shower, a pulsar harmonic or a fluorescent spot is a few strong coefficients sitting on a dense noise floor. Rotating that mixture smears the few strong coefficients across every coordinate, and the coarse grid then has to be scaled to the full signal energy, including the part that was concentrated in the spikes.
We propose a hybrid code built for fixed-latency hardware. Each block of the stream is encoded as exactly $k$ "spike" slots, each an index plus a moderately precise coefficient, chosen by a fixed-depth sorting network. What remains after removing the spikes is a dense residual. We pass it through a multiplier-free Hadamard rotation and quantize it at 1–2 bits with a single shared scale. Because $k$ is constant, the encoder has a fixed number of pipeline stages and a constant initiation interval, meaning it accepts a new block at a steady rate. It has none of the variable-length outlier lists that make existing sparse-plus-quantized schemes awkward on FPGAs. The central claim is that under a fixed total bit budget, the hybrid strictly beats both pure rotation-then-quantize and pure top-$k$ in the regime where signal energy is concentrated but the floor is not negligible. It does so while meeting nanosecond deadlines. We also claim that this crossover can be predicted in advance from two measurable quantities of the stream.
Core Claim
Consider a block of $n$ values with total bit budget $B$. The hybrid spends $B = k(\log_2 n + b_c) + n\,b_r$ bits. Here $k$ is the number of spike slots, $\log_2 n$ bits name each slot's position, $b_c$ is the coefficient precision, and $b_r$ is the bits per value for the rotated residual. After a Hadamard rotation the residual is close to Gaussian. Standard rate–distortion behavior for Gaussian sources then gives residual distortion of roughly $c\,E_r\,2^{-2b_r}$, where $E_r$ is the residual energy and $c$ a constant set by the quantizer. Rotating the whole block instead, as incoherence processing does, gives roughly $c\,(E_s+E_r)\,2^{-2b}$ with $b = B/n$, where $E_s$ is the energy captured by the top $k$ coefficients. The hybrid therefore wins when
$$\frac{E_s + E_r}{E_r} > 2^{\,2\,(b_r' )},\qquad b_r' = \frac{k(\log_2 n + b_c)}{n},$$
meaning the energy ratio removed by the spikes must exceed the precision the residual gives up to pay for the slots. The spike coefficients themselves contribute only their own small $b_c$-bit error. The claim is falsifiable. For a given detector we can measure the "peakiness" ratio $(E_s+E_r)/E_r$ from calibration data, predict which of the three codes wins at each budget, and check that prediction.
Relation to Prior Work
Several papers motivate the problem. Bits missing: Finding exotic pulsars using bfloat16 on NVIDIA GPUs shows that a radio pulsar search tolerates dropping from 32-bit to bfloat16 in the stages after the first FFT, gaining a 1.6x speedup, but only if dynamic range is kept via an 8-bit exponent. It never goes below 16 bits. FastML Science Benchmarks defines the collider calorimeter compression task: 48 trigger cells squeezed into 144 or 48 bits within 100 ns. It solves the task with a black-box autoencoder and uses no sparsity prior. hls4ml shows that 6-bit quantization-aware training with pruning holds accuracy on FPGA classifiers. The LLM evaluations collected under "Evaluating Quantized Large Language Models" (e.g.; Cherry on Top: Parameter Heterogeneity and Quantization in Large Language Models) document the cliff below 3 bits.
Three papers supply parts of the method and also define the novelty boundary. QuIP rotates static weight matrices with random orthogonal transforms before 2-bit rounding. On Llama 2 70B this rescues WikiText2 perplexity from 123.91 to 6.33. We reuse the rotation, but only on the residual left after the spikes are removed, because rotating a spiky signal destroys exactly the structure we want to keep. SpQR keeps under 1% of weights as full-precision outliers and quantizes the rest at 3–4 bits. AnchorKV stores anchor tokens in bfloat16 and 2-bit residual corrections. These are the closest relatives: both split a signal into a sparse high-precision part and a dense low-precision part. They differ from our proposal in three ways. They compress static weights or attention caches rather than live sensor streams. Their outlier count varies with the data, which forces separate sparse kernels (SpQR) or adds 1.12–1.30x prefill time (AnchorKV). They pair the split with no rotation (SpQR) or with attention-specific projections (AnchorKV).
Our slot count is fixed by design. That is what makes the code synthesizable at a constant initiation interval, and it resolves the tension between outlier isolation and fixed-pipeline hardware. Lexico shows that sparse coding over learned dictionaries beats 2-bit quantization for caches: 48.75% versus 40.56% on GSM8K for Llama-3-8B. Its orthogonal matching pursuit, a greedy iterative search that picks dictionary atoms one at a time, is too slow for streaming. We avoid pursuit entirely by using a complete orthonormal basis, so exact top-$k$ selection is a sort. We accept the loss of overcompleteness as a cost to measure. On Sparsity and Overcompleteness in Image Models found the optimal dictionary to be only modestly overcomplete, which suggests the cost is small.
Robustness-aware 2-bit quantization constrains the design. Its heavy-tailed Student-t modeling and shift-add grids work for classifier weights on FPGA, but they use fixed grids that do not separate out rare large values. We treat it as a hardware-cost reference, not a competitor on streams.
Research Program
The work has three parts. The first is the encoder. It consists of a fixed-$k$ top-magnitude selector implemented as a bitonic sorting network over $n$ values; slot coefficients at $b_c$ bits with a shared exponent; subtraction of the spikes; an $n$-point Hadamard transform, which has $\log_2 n$ add/subtract stages and no multipliers; and uniform $b_r$-bit residual quantization with one scale per stream, recalibrated slowly. Decoding reverses these steps linearly. We will write it in HLS C++ inside the hls4ml flow and report latency, initiation interval and resource use.
The second part is choosing the basis. The identity basis suits calorimeter cells and complex FFT spectra. A learned orthogonal basis, trained offline as a product of Givens rotations so that it stays orthonormal and fast, suits blocks where spikes are spatially extended. The third part is the allocation rule: the design equation above, fitted on calibration data, picks $(k, b_c, b_r)$ per stream type. Allocations are fixed at synthesis time, so latency never depends on the data.
Baselines
The naive reference is uniform per-block quantization at the same bits per value, with the block scale set by the maximum magnitude. This is the "just downcast" option. Everything should beat it once the budget reaches 2 bits or less.
The strongest existing method for the calorimeter task is the FastML/hls4ml CNN autoencoder, which achieves 25 ns latency at 60 mW in ASIC simulation. Its reconstruction quality is not reported in the file: there is no reported result on Lacuna for the HGCal autoencoder's reconstruction metric. We will therefore retrain it ourselves on the same data split. It gets the same 48- and 144-bit payloads, a hyperparameter search budget matching the one we give our allocation fit, and the benchmark's own reconstruction metric alongside ours. Beating it means lower reconstruction error on the highest-energy cells at equal payload and equal or lower latency. Alternatively, if in-distribution error ties, it means clearly better error under detector drift.
For the pulsar setting, the strongest baseline is the bfloat16 FDAS pipeline from Bits missing. In that pipeline 97.53% of 14,971 bright single-precision peaks land in the same frequency and acceleration bin, and all peaks fall within 3% of single-precision SNR. Beating it means matching those two numbers while storing the post-FFT spectrum at about 4 bits per complex value instead of 32.
The key ablations isolate each ingredient under identical budgets. The first is QuIP-style rotation plus uniform quantization with no spike slots. The second is top-$k$ only, with the whole budget spent on slots and no residual. The third is the hybrid without residual rotation. The fourth is a variable-count threshold version resembling SpQR, which shows what the fixed-$k$ constraint costs in accuracy and saves in latency.
Why not just…?
Why not just train a quantized autoencoder end to end, as FastML already does? This is the real threat. A learned encoder at 48 bits may match our in-distribution error, because it can learn the spike-plus-floor structure implicitly. Our bet rests on two things an autoencoder does not guarantee. The first is a hard error bound on the $k$ largest cells, which carry the physics. The second is behavior when the noise floor or gain drifts, since our only adaptive quantity is one scale, while the autoencoder must be retrained and resynthesized. The drift experiment below decides this. If the autoencoder wins there too, the contribution shrinks to an interpretable, analyzable encoder that can serve as the quantization layer inside autoencoders. We would report that outcome as such.
Why not just zero-suppress, sending only cells above a threshold as detectors already do? Thresholding produces a variable number of outputs, which breaks fixed payloads and fixed latency. It also discards the floor entirely, and the floor carries information in pile-up-dominated or noise-limited regimes such as weak pulsars. The top-$k$-only ablation is exactly this idea with a fixed size. The design equation predicts it wins only when $E_r$ is negligible, and the synthetic sweep tests that prediction.
Why not just rotate and quantize, as QuIP does? For a spiky block, rotation forces a grid scaled to the total energy, so the residual precision is wasted on energy that $k$ indices could have captured nearly exactly. That is the core claim, and the rotation-only ablation is its direct test.
Why not just use bfloat16? Bits missing already shows bfloat16 works, but it only halves memory traffic. At SKA rates, that does not fix the gap between data rate and bandwidth. We match its fidelity numbers at a quarter of its bits.
Proposed Experiments
The first experiment is a synthetic crossover test. We generate blocks with $n \in \{16, 48, 256\}$, a controlled number of spikes and a controlled spike-to-floor energy ratio. We sweep budgets from 1 to 3 bits per value. The prediction: the design equation places the winner among rotation-only, top-$k$-only and hybrid correctly in most grid cells. A failed prediction would mean the Gaussian-residual approximation is wrong for this setting.
The second experiment uses the calorimeter benchmark from FastML Science Benchmarks. At 48 and 144 bits we report total reconstruction error, error on the three hottest cells, and synthesized latency against the 100 ns budget, comparing the hybrid against all baselines. The drift test follows: we train or calibrate on nominal conditions, then evaluate with a scaled noise floor and a shifted gain. We compare three things: the autoencoder unchanged, the autoencoder retrained, and our encoder with only its scale updated.
The third experiment follows the pulsar setup of Bits missing. We take the roughly 20,000 synthetic SIGPROC files and the GMRT observation of PSR J1544+4937. We keep the first real-to-complex FFT in 32-bit and encode its output spectrum with the hybrid at about 4 bits per complex value. We then run the matched-filter search and report same-bin agreement and SNR deviation relative to single precision, against the 97.53% and 3% figures. We also check whether PSR J1544+4937 is still recovered.
The decisive checks are three. The hybrid must beat rotation-only at 2 bits on calorimeter hot-cell error. It must match bfloat16 bin agreement at 4 bits on pulsars. And the fixed-$k$ encoder must synthesize with a constant initiation interval under 100 ns. If the first check fails, the central thesis is wrong. If only the third fails, the idea survives but is limited to GPU-scale latencies.
References
- Bits Missing: Finding Exotic Pulsars Using bfloat16 on NVIDIA GPUs
- FastML Science Benchmarks: Accelerating Real-Time Scientific Edge Machine Learning
- hls4ml: An Open-Source Co-Design Workflow to Empower Scientific Low-Power Machine Learning Devices
- Cherry on Top: Parameter Heterogeneity and Quantization in Large Language Models
- QuIP: 2-Bit Quantization of Large Language Models With Guarantees
- SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression
- AnchorKV: Anchor-Residual KV Cache Compression
- Lexico: Extreme KV Cache Compression via Sparse Coding over Universal Dictionaries
- On Sparsity and Overcompleteness in Image Models
- Robustness-aware 2-bit quantization with real-time performance for neural network