Shared-Prediction Sparse Coding: Sending LLM Activations in Short Packets

When a language model is split between a phone and a server, the phone must send the hidden state for every generated token over the radio. If each token has to arrive within a millisecond with near-certain reliability, that hidden state has to fit in a short packet: a few hundred channel uses, roughly 20 to 100 bytes after error-correction overhead. A LLaMA-class hidden state has 4096 numbers. The usual way to shrink it is activation sparsity, which keeps only the few largest entries for each token and drops the rest. The catch is that the receiver then has to be told which entries survived. Keeping 64 of 4096 entries costs about $\log_2\binom{4096}{64}\approx 476$ bits of position information, compared with 256 bits for the 64 values at 4 bits each. Most of the packet ends up describing where the numbers are rather than what they are.

We propose that the sender and the receiver each run the same small predictor, trained offline, on the hidden states the server has already successfully received for earlier tokens. The predictor ranks the coordinates of the current token by expected importance, so the sender can transmit values in the predicted order without sending any indices. It then adds a short correction for coordinates the predictor missed. How many coordinates go into each packet is set token by token from the current radio conditions, using finite-blocklength rate formulas. If this works, activation sparsity keeps its per-token adaptivity without paying the signaling cost that makes it useless on short packets.

Core Claim

Our hypothesis is that, at equal bits per token, coding hidden states in an order predicted from shared history beats two alternatives: a fixed linear transform (a PCA basis with bit allocation), and per-token magnitude top-k with entropy-coded indices. We further hypothesize that the gain comes from the temporal predictability of which coordinates are large. The design contains the fixed transform as a special case, the one where the predictor outputs a constant ranking. So the method can at worst tie that baseline, and the size of the gap is exactly what we measure. We also claim that an ordered, index-free payload degrades gracefully when bits are lost, which lets it use unequal error protection. Index-based payloads cannot, because one corrupted position bit misplaces a value.

Where This Sits

The sparsity side supplies the observations that make the idea plausible.

  • Evidence that heavy-hitter sets are concentrated. MoEfication showed that ReLU Transformers activate small, clustered neuron subsets per token. SoLA found that about 15% of neurons carry about 95% of activation norm in LLaMA-2. This suggests the important coordinates are concentrated and partly stable, which is the structure a predictor could exploit.
  • Methods we reuse. TEAL zeroes low-magnitude activations using output-reconstruction error. CHESS sets per-channel thresholds by the channel's effect on the next weight matrix. We adopt their weighted, output-aware notion of distortion.
  • The hybrid design. R-Sparse routes heavy-hitter channels through a sparse path and the rest through a low-rank path. That hybrid is one of our baselines.
  • How our setting differs. All of these methods aim to skip memory reads on a local GPU or CPU. There, knowing which entries are nonzero is free, so none of them faces the index-signaling problem that dominates on a radio link.

The closest bridge between the two areas is Comet. It uses a low-rank predictor to forecast active neurons and cut communication in secure multi-party inference. The differences are specific:

  • Comet predicts from the current pre-activation so the parties can skip computation.
  • It runs over reliable wired links.
  • It relies on ReLU-fication.

We predict from shared past state so that a receiver who has not yet seen the current token can reconstruct the ordering. We also operate on smooth-activation models over a lossy fading link, where a lost packet desynchronizes the two predictors.

SIMBA benchmarks split inference, but it treats the link as an error-free pipe and uses no sparsity.

On the communication side, several papers constrain the design:

  • On Finite Blocklength Lossy Source Coding shows that separating source and channel coding loses second-order efficiency at short blocklengths.
  • Scarlett et al. obtain gains from unequal error protection across source classes, but only for synthetic memoryless sources.
  • Xu et al. show that capacity-based power allocation for short packets drives packet error to 50% at a $5\times10^{-4}$ target. This is why our payload sizing must use the finite-blocklength rate rather than Shannon capacity.
  • She et al. argue that coding error, queueing violation and dropping should be balanced at the same order of magnitude. We extend that balance with a fourth term, inference distortion.

Research Program

Setting. A client runs the first $l$ blocks of a decoder model. The split is required, not optional: the client holds private or personalized early layers (for example, a local adapter). Each decoded token's residual-stream vector $h_t\in\mathbb{R}^{4096}$ is sent to the server within $B$ mini-slots of $n$ channel uses each.

Bit budget. Given the instantaneous signal-to-noise ratio $\gamma_t$ and a target packet error $\epsilon$, the number of payload bits the link can carry follows the normal approximation:

$$k_t=nB\,C(\gamma_t)-\sqrt{nB\,V(\gamma_t)}\,Q^{-1}(\epsilon),$$

Here:

  • $C$ is the Gaussian-channel capacity.
  • $V$ is the channel dispersion, the variance term that penalizes short blocks.
  • $Q^{-1}$ is the inverse Gaussian tail function.

The $\sqrt{nB}$ scaling comes from the central-limit behavior of the information density at finite blocklength. $k_t$ therefore fluctuates with fading.

Encoder. We first compute an output-weighted orthonormal basis offline on C4. It is the SVD of the calibration hidden states, whitened by the next block's input projections, following the output-aware distortion of CHESS and TEAL. Hidden states are expressed in this basis. The known massive outlier coordinates are kept as explicit basis directions.

A small predictor $g$ takes the last $w$ successfully received reconstructions $\hat h_{t-w},\dots,\hat h_{t-1}$. It outputs a ranking and a scale estimate over the basis coefficients. Both ends run $g$ identically.

For each token, the encoder sends:

  • the top-$m_t$ coefficients in predicted order, scalar-quantized with per-rank step sizes;
  • a short correction block that swaps in up to $c$ coefficients the predictor under-ranked, and which does carry explicit indices.

The encoder chooses $m_t$ and $c$ to fill $k_t$.

Error protection. The first coefficients in the predicted order matter most, so the payload is naturally layered. We protect the head of the payload with a lower error target than the tail, which is a practical instance of the unequal error protection studied by Scarlett et al.

Handling packet loss. If a packet is lost, both sides detect it through the acknowledgment. They then drop that token from the predictor's history and fall back to the static basis ordering until $w$ clean tokens accumulate. Resynchronization therefore needs no extra signaling.

What we train. Only the predictor $g$ is trained: a two-layer network over a compressed history, trained on FineWeb (HuggingFaceFW/fineweb) to minimize output-weighted reconstruction error under a sampled bit budget. The language model stays frozen.

What we analyze. Beyond the end-to-end system, we analyze the information-theoretic quantity that decides whether the idea can work. This is the conditional entropy of the current token's top-$m$ coefficient set given the past $w$ reconstructions, compared with its unconditional entropy. The gap between the two is the largest index saving any shared predictor can deliver. We report it per layer and per model.

Baselines

Uncompressed reference and the dense-accuracy anchor. - The naive reference is the unsplit model's accuracy and perplexity with an ideal link. - To anchor how much accuracy local sparsity alone costs, TEAL reports a LLaMA-3-8B downstream-suite average of 68.07 dense and 66.21 at 40% model-wide sparsity (TEAL). - CHESS reports 67.07% average accuracy at about 68% parameter activation on its eight-benchmark suite (CHESS). - These numbers are for compute sparsity, not bit budgets. We found no reported result on Lacuna for LLM accuracy under a per-token bit budget in split inference, so all bit-budget curves for the baselines below are produced by us.

Fixed transform coding. This is the strongest competitor. It uses the same output-weighted PCA basis, sends the top-$m_t$ coefficients in a fixed order, and allocates bits by reverse water-filling on calibration variances. It is exactly our method with a constant predictor, so the comparison isolates the value of conditioning on shared history. It gets the same calibration data and the same quantizer search budget. We count our method as beating it if we get a lower perplexity at every budget between 160 and 2000 bits, and at least one point of downstream-suite accuracy at budgets of 400 bits or fewer.

Per-token magnitude top-k. This is TEAL-style selection in the same basis and in the raw coordinates. The indices are entropy-coded with an adaptive arithmetic coder conditioned on per-coordinate activation frequencies, which is the strongest cheap way to code positions.

R-Sparse-style hybrid. Following R-Sparse, this sends heavy-hitter coordinates with indices plus a rank-$r$ sketch of the remainder. We tune the split between $m$ and $r$ with the same search budget as our quantizer.

Ablations. Three ablations isolate our ingredients:

  • removing the correction block;
  • replacing the learned predictor with "copy the previous token's ranking";
  • replacing unequal error protection with uniform protection.

Wireless side. All schemes use identical finite-blocklength sizing. As a sanity check we also run capacity-based sizing. We expect it to reproduce the collapse Xu et al. report, packet error rising to 50% at a $5\times10^{-4}$ target (Xu et al.).

Why not just…?

Why not just use the fixed PCA transform? This is the real threat. If the large coefficients of a hidden state are not predictable from recent tokens, the fixed transform captures essentially everything, and our predictor adds cost for nothing. The conditional-entropy measurement settles this before any system is built. If the past explains less than about 10% of the entropy of the top-$m$ set at early layers, we abandon shared prediction and report transform coding with unequal error protection as the answer. That would still be a useful negative result for split inference.

Why not just entropy-code the top-k indices well? Entropy coding recovers only the non-uniformity of how often each coordinate is active. It cannot exploit token-to-token correlation unless it conditions on history. Once it does, it becomes a weaker version of our predictor. The magnitude top-k baseline with a frequency-adaptive coder measures exactly this gap. Index bits are also fragile under channel errors, which the uniform-protection comparison exposes.

Why not just send the token instead of its hidden state? For plain text offloading with no private client computation, a token ID costs about 17 bits and beats every activation scheme. We say so plainly and restrict the claim to splits that exist because the client holds weights the server must not see. We still report the token-ID cost as a lower reference on every plot. We do not claim privacy: aggressively compressed early-layer states may be easy to invert. We will run SIMBA-style reconstruction attacks as a diagnostic only.

Why not just train an end-to-end learned joint source-channel encoder? A learned encoder that maps activations straight to channel symbols could in principle beat any digital pipeline, and finite-blocklength theory favors joint coding (Kostina). But it requires retraining per channel model and signal-to-noise range, and it offers no decoding-error target that can be certified at $10^{-5}$. Our design keeps a standard short-packet code with a known error probability. We treat learned joint coding as follow-up work, not a competitor we can fairly tune here.

Proposed Experiments

Models, data and splits. We use LLaMA-3-8B and Mistral-7B, the models TEAL and R-Sparse evaluate on, split after blocks 2, 4 and 8. Language modeling is measured as WikiText-2 perplexity. The downstream suite is ARC-Easy and ARC-Challenge, HellaSwag, BoolQ, PIQA, Winogrande and MMLU. Calibration uses C4.

Channel. The channel follows the Rayleigh block-fading short-packet setup of Mili et al.. They report that throughput is extremely sensitive to codeword length below 500 channel uses, and a 66% gain from finite-blocklength-aware optimization at $L=100$. We sweep:

  • $n \in \{100, 200, 500\}$;
  • $B \in \{1, 4, 16\}$;
  • $\epsilon \in \{10^{-3}, 5\times10^{-4}, 10^{-5}\}$;
  • average signal-to-noise ratio from 0 to 20 dB.

Metrics. - Perplexity and accuracy against average bits per token. - A token-failure rate: the fraction of tokens whose next-token distribution differs from the dense model by KL divergence above a threshold, counting both packet loss and compression distortion. - The 99.999th-percentile per-token distortion, since reliability targets concern tails, not averages. - Energy per token under the Xu et al. convex scheduler.

Decisive checks, in order. First, the conditional-entropy measurement. It is cheap and can kill the idea. Second, at matched bit budgets, the comparison against fixed transform coding, which is the result the proposal stands or falls on. Third, whether the ordered payload with unequal error protection lowers tail distortion at $\epsilon=10^{-5}$ relative to index-based schemes. Finally, whether adapting $m_t$ to fading keeps the model on the safe side of the sparsity cliff that TEAL and CHESS report beyond 60–70% sparsity. The alternative is a fixed average-sized payload, and the test is how often deep fades push that fixed payload past the cliff.