Learned compressors turn an image or video into a bitstream using probabilities that a neural network predicts. The decoder must recompute those probabilities bit for bit, and in practice it often cannot. Different GPUs and CPUs add up the same numbers in different orders and round differently, so a decoder on a laptop may compute 0.4000002 where the encoder on a server computed 0.4000001. That tiny gap is enough to corrupt everything after it. The standard fix is to rewrite the network in integer arithmetic. This works, but it costs compression quality and, on float-optimized accelerators, speed.
We propose a different route. Floating-point addition is order-independent whenever no rounding happens at all, and no rounding happens when every partial sum is an integer small enough to fit in the format's significand (the bits that hold a number's digits). How large the partial sums of a dot product can get depends mostly on how many terms it has. A sparse layer in which every output neuron reads exactly $k$ inputs therefore gives a hard, uniform guarantee. Its integer-valued weights and activations can run on ordinary fp16/fp32 tensor cores and still produce bit-identical results on any hardware and in any summation order. This "fixed in-degree" pattern is borrowed from the regular-degree sparse graphs that theory finds optimal for associative memory. The bits that sparsity saves in the accumulator can then be spent on finer weight and activation precision. That is the mechanism we expect to recover the quality loss of integerized codecs without running slower integer kernels.
Core Claim
Let $p$ be the number of significand bits of the accumulator: 24 for fp32 and 11 for fp16. Every integer up to $2^p$ in magnitude is then exactly representable. Consider output neuron $i$ with integer weights $w_{ij}$, an integer bias $b_i$, and integer activations bounded by $A_{\max}$. Each partial sum is bounded by the sum of the absolute values of all its terms, so the condition below makes the computation exact and order-free:
$$A_{\max}\,\lVert w_i\rVert_1 + |b_i| \le 2^{p}.$$
Under this condition, reduction order, use or non-use of fused multiply-add, and vendor library choice cannot change the result. With $b_w$-bit signed weights, $b_a$-bit activations and in-degree $k$, the condition reduces to roughly $b_w + b_a + \log_2 k \le p + 1$.
A dense 3×3 convolution with 192 input channels has $k = 1728$ and uses about 11 of the 24 bits on accumulation headroom alone. A fixed-degree layer with $k = 64$ uses 6. The claim is that, at matched multiply count and with cross-platform exactness enforced, spending the freed bits on precision gives better rate-distortion than dense integer networks. In the fp16-accumulate regime, where $p = 11$ and dense layers simply cannot be made exact at useful widths, fixed in-degree is what makes exact float execution possible at all.
Why Fixed In-Degree, and Where It Comes From
Two biological-connectivity results motivate the constraint. Graham et al. show that sparse associative networks at 1–10% connectivity maximize information stored per connection. Zhang and Chen prove that a Delta degree distribution, where every neuron has the same degree, beats binomial and power-law graphs for retrieval, because low-degree neurons drown in cross-talk. We do not claim codecs are attractor networks. We use these results as evidence that a uniform-degree constraint need not cost much expressivity, and our experiments test that directly.
The same constraint has a hardware payoff. Every row has the same length, so the layer stores as fixed-width index and value arrays with no load imbalance. We also let groups of $g$ output neurons share one input set, mirroring the locally dense, globally sparse modules of CircuitNet. Each group then becomes a gather followed by a small dense matrix multiply that tensor cores and CPU vector units can run. This is the regularity that brain-inspired dynamic sparse training such as CHTs lacks; that paper concedes its unstructured masks give no wall-clock gains on GPUs.
Research Program
The work has three parts.
The first is a training recipe. We train with quantization awareness on integer grids that are represented as fp16 values. A learned per-output top-$k$ mask enforces the in-degree. To control mask flicker during training we reuse the refined straight-through estimator of Zhou et al.. We add a projection step that clips each row's $\ell_1$ norm to the exactness budget. Rescaling between layers uses power-of-two shifts followed by rounding, both of which are exact in floating point. Nonlinear maps on the probability path, such as producing the scale $\sigma$, are replaced by integer-indexed lookup tables, following the integerized hyperprior designs of Ballé et al..
The second is the exactness analysis. Deterministic rounding bounds grow as $n u$ with layer width $n$ and unit roundoff $u$, per Deterministic and probabilistic rounding error analysis. Those bounds describe error size, whereas our condition makes the error identically zero. We will characterize which accelerator accumulators actually honor $p$ significand bits, since some tensor-core paths truncate, and set $p$ per device class accordingly.
The third is kernels. We will build a grouped gather-then-GEMM kernel for NVIDIA and AMD GPUs and for x86 and ARM vector units, and benchmark it against dense fp16 and int16 execution. DCVC-RT (Jia et al.) found operational overheads such as memory traffic and kernel launches dominating FLOPs, so wall-clock time is the metric, not multiply counts.
Baselines
The naive reference is the floating-point hyperprior codec run unmodified across devices. On Kodak it fails to decode 34% to 78% of the time between Xeon CPUs and Titan X GPUs, as reported by Ballé et al.. We re-measure this on our device pool to confirm the failure mode is present.
The strongest determinism baseline for images is the integer network of Ballé et al., which reports a 0% failure rate. For video it is the statically 8-bit quantized hyperprior of Scale-Space Flow from Conceição et al.. That method reduces cross-machine PSNR divergence below 0.001 dB at a BD-rate cost of 9.93% on UVG and 9.02% on HEVC-B. BD-rate is the average bitrate change needed to reach equal quality, so lower is better. Beating these baselines means matching 0% failure and bit-identical probability tables while cutting the BD-rate penalty well below 9–10% on the same sequences.
For speed, the dense int16 path of DCVC-RT is the reference. That paper reports it runs slower than fp16 but gives no number for the gap, so we measure it ourselves. All determinism baselines get the same architecture skeleton, training data, step budget and quantization-aware fine-tuning schedule.
Two ablations isolate the key ingredient. In the first, a dense exact-in-float network uses the same accumulator but lowers $b_w$ and $b_a$ until the budget holds. This tests whether sparsity-for-precision helps, or whether low bit width alone suffices. In the second, a variable-degree sparse network has the same total nonzeros and the same per-row $\ell_1$ budget. This separates the benefit of uniform degree, both in quality and in kernel regularity, from the benefit of sparsity per se.
Why not just…?
Why not just integerize the dense network? It works for determinism. The cost is the 9–10% BD-rate penalty reported by Conceição et al. and the int16 slowdown in DCVC-RT. The dense exact-in-float ablation is the real threat here, because it gets float-unit speed with no sparsity at all. If it matches our rate-distortion at equal multiplies, the sparsity half of the proposal fails and we would report that plainly.
Why not just use mismatch-tolerant entropy coding? PMATIC absorbs bounded logit mismatch for 1D language-model token streams by sending helper bits. It has not been shown on spatial image latents. It also cannot stop drift that compounds through reconstructed reference frames, which the $O(L^T\epsilon)$ growth analysis of CSSC describes and Conceição et al. observe as breakdown by frame 5. Our approach costs zero helper bits. We will nonetheless run a PMATIC-style tolerance band on the image codec as a bitrate-overhead comparison.
Why not just use 2:4 sparsity on Sparse Tensor Cores? 2:4 is in fact a fixed-degree pattern. At 50% density, however, it frees only one bit of accumulator headroom, its acceleration exists only on Ampere-class NVIDIA hardware (Zhou et al.), and run in float it is no more deterministic than dense. We include 2:4 as a special case of our constraint and report where it sits on the rate-distortion curve.
Why not just turn on deterministic kernel modes? Those modes fix the reduction order within one library on one device. They do nothing across vendors, instruction sets or fused-multiply-add policies, and the failures documented by Ballé et al. are exactly that cross-device case. Our device-pool test includes this configuration to show it directly.
Proposed Experiments
The first check is exactness, on images. We train a hyperprior codec in the style of Ballé et al. under our constraint and encode Kodak, Tecnick and CLIC 2020 Professional on one device. We decode on NVIDIA and AMD GPUs, an x86 CPU and an ARM CPU, under both fp32 and fp16 accumulation. The required outcome is 0% decoding failure and identical hashes of every probability table, with the float baseline reproducing its high failure rate on the same pool.
The second check is rate-distortion at equal multiplies. Across $k$ from 16 to 512 and the corresponding $(b_w, b_a)$ splits, we compare PSNR-bitrate curves against the dense integer network and the two ablations. We expect a quality optimum at intermediate $k$, which Graham et al. would place around 1–10% connectivity. Gains should be largest under fp16 accumulation.
The third check is temporal drift. We apply the constrained layers to the Scale-Space Flow hyperprior path and evaluate on UVG and HEVC-B, reporting cross-machine PSNR divergence over long sequences. The decisive comparison is BD-rate cost against the 9.93% and 9.02% figures of Conceição et al..
The fourth check is wall-clock time. We measure decode latency at 1080p and 4K for our grouped kernel against dense fp16, dense int16 and tile-wise sparse execution. For context, Guo et al. report tile-wise sparse execution 1.95× faster on V100 tensor cores. The proposal succeeds if exactness comes at no more latency than dense fp16 on at least the GPU and ARM targets.
References
- Capacity and Information Efficiency of a Brain-like Associative Net
- Transient dynamics of sparsely connected Hopfield neural networks with arbitrary degree distributions
- CircuitNet: A Generic Neural Network to Realize Universal Circuit Motif Modeling
- Brain network science modelling of sparse neural networks enables Transformers and LLMs to perform as fully connected
- Learning N: M Fine-grained Structured Sparse Neural Networks From Scratch
- Integer Networks for Data Compression with Latent-Variable Models
- Deterministic and probabilistic rounding error analysis of neural networks in floating-point arithmetic
- Towards Practical Real-Time Neural Video Compression
- Cross-Platform Neural Video Coding: A Case Study
- Efficient Mismatch-Tolerant Coding for Model-Driven Compression
- Causal State Compression for Long-Horizon Video World Models: A Bounded-Drift Theory and Efficient Architecture
- Accelerating sparse DNN models without hardware-support via tile-wise sparsity