Linear recurrent sequence models, or state space models (SSMs), such as Mamba process a sequence by updating a fixed-size hidden state one step at a time. On paper they are cheaper than Transformers. On real accelerators they often run slower, because each step spends most of its time moving that hidden state between slow off-chip memory and fast on-chip memory, and comparatively little time computing. Storing the state in 4 bits instead of 16 would cut that traffic about fourfold. Today this fails. The best post-training recipe stops at 8 bits, and pushing to 4 bits collapses the model, because rounding errors fed into the recurrence pile up over thousands of steps.
This note proposes a specific reason for the collapse and a fix that follows from it. Each channel of a diagonal SSM forgets at its own rate. A channel that forgets slowly amplifies rounding noise by a factor that can be predicted from that rate, much as a Kalman filter's error grows when it keeps predicting without fresh observations. So precision should go where memory is long. The method assigns a small set of hardware-native formats (INT4 or INT8) to channels according to their noise-amplification factor. It also lets each channel's quantization range shrink and grow along with a predictable running bound on the state, a device borrowed from adaptive quantization in distributed optimization. We test the method where long memory matters most and failures are physical rather than a perplexity uptick: SSMs used as predictors and controllers in feedback loops whose sensor packets arrive late or not at all.
Core Claim
Take a diagonal recurrence $h_t = a_t \odot h_{t-1} + b_t$, where $a_t \in (0,1)^N$ are per-channel decay factors and $b_t$ is the input-driven update. If the stored state is re-quantized every step with step size $\delta_i$ in channel $i$, the injected noise has variance about $\delta_i^2/12$, and its accumulated stationary variance is
$$\sigma_i^2 \approx \frac{\delta_i^2}{12}\cdot\frac{1}{1-\bar a_i^2},$$
where $\bar a_i$ is the channel's typical effective decay. This is the standard variance of a first-order autoregressive process driven by white noise. Two consequences follow. First, uniform 4-bit quantization fails mainly in the few channels with $\bar a_i$ near one, and not across the board. Second, choosing bits to equalize $\sigma_i^2$ under a total memory budget gives roughly $b_i \approx \log_2(R_i/\epsilon) + \tfrac12\log_2\!\big(1/(1-\bar a_i^2)\big)$, where $R_i$ is the channel's range and $\epsilon$ a target error. This is a classical rate-allocation rule, rounded to the formats the hardware actually supports.
The claim is falsifiable. If 4-bit failure is not concentrated in long-memory channels, or if gain-ordered allocation does no better than allocation by activation magnitude, the idea is wrong.
How This Relates to Prior Work
Several papers motivate the gap. Avenstrup et al. show that SSM implementations are dominated by moving hidden states between off-chip and on-chip memory. Huang et al. and Liu argue that software stacks are tuned for dense matrix multiplication, which disadvantages sequential scans. This makes state bytes, rather than FLOPs, the quantity to cut.
The closest existing work is Quamba, which builds an 8-bit post-training recipe from percentile clipping, Hadamard rotation of scan outputs, and fused scan kernels. It reaches a 0.9% average zero-shot drop on Mamba 2.8B and 1.72× lower generation latency on an Orin Nano. Quamba itself reports that sensitivity to errors compounding through the recurrence is the obstacle, and that 4-bit weight-activation quantization collapses. SSDi8 and QMamba also use static, uniform low-bit formats. We differ in two ways. We target the stored recurrent state specifically, and we allocate precision non-uniformly by a quantity, decay-induced noise gain, that none of these papers uses.
The theory that makes per-channel allocation natural comes from On Structured State-Space Duality. It shows that diagonal-transition SSMs carry $N$ distinct decay rates at the same asymptotic cost as scalar ones. For scalar-decay models such as Mamba-2, our allocation operates per head instead of per channel.
The range-tracking mechanism comes from adaptive quantization. Thanou shrinks the quantization interval as consensus states settle, which prevents noise accumulation. Maguluri et al. prove that matching the interval's decay to the step size removes the error floor. Dyna-COLD scales the quantizer input dynamically so that even 1-bit compressors converge linearly. We reuse the principle that the quantizer range should follow a predictable bound on the signal. Here that bound is the running envelope $s_t = a_t s_{t-1} + |b_t|$, computable in-kernel at negligible cost.
We also take a constraint from this line of work. Per-step bit-switching is unfriendly to fixed-format hardware. We therefore fix each channel's format at calibration time and adapt only the scale, which is a single multiply.
The analysis lens and the application come from control. Sinopoli et al. show that a Kalman filter's error stays bounded only if observations arrive with probability above a critical value $\lambda_c$. A selective SSM whose input gate is nearly closed resembles a filter running without observations. A sensor stream with dropouts lengthens exactly those stretches. Molnár et al. show that delayed loops need forward prediction over the delay, which is a long-memory task. Flash STU evaluates SSMs on linear dynamical systems and MuJoCo control, but only with ideal full-precision, delay-free feedback. None of these papers quantizes the model that sits inside the loop.
Research Program
The first stage is diagnostic. On pretrained Mamba and Mamba-2 checkpoints, we log the effective per-channel decay distribution on calibration data. We then quantize the state of one channel group at a time to 4 bits and measure the damage. If the first-order picture holds, damage should rank with $1/(1-\bar a_i^2)$ more closely than with activation magnitude.
The second stage builds the method. We compute gain scores from calibration statistics and fill an INT8 slot for the highest-gain channels until a byte budget is met; the rest get INT4. Channels are permuted so that each precision group is contiguous in memory. Per-channel envelope-tracked scales are fused into the scan kernel. This is engineering on top of Quamba-style fusion, not a new kernel design.
The third stage asks whether the allocation holds under distribution shift. Packet dropouts and delays change how often the input gate closes, so a static allocation calibrated on clean streams could be wrong exactly when it matters. We measure this directly. If it is wrong, we add a cheap online correction that promotes a channel group when its envelope exceeds its calibrated range.
Baselines
The do-nothing reference keeps the recurrent state in FP16 and applies Quamba's 8-bit recipe elsewhere. This defines the accuracy ceiling and the memory-traffic floor we aim to beat.
The strongest existing method is Quamba at INT8. Its reported figures are a 0.9% average zero-shot accuracy drop on Mamba 2.8B and 1.72× lower generation latency than FP16 on Orin Nano 8G. For Quamba at 4 bits, the file reports collapse but gives no number, so no reported result was found on Lacuna for a 4-bit Quamba accuracy. On control benchmarks, Flash STU reports MuJoCo Ant validation loss of 0.0139 for Mamba-2 and 0.0092 for STU-T at full precision. No reported result was found on Lacuna for any quantized model on these tasks, so we will establish quantized baselines ourselves.
All variants will use the same checkpoints, the same 512-sequence calibration set, the same fused-kernel code path, and the same hardware. Only the state-storage policy will differ.
Beating Quamba means matching its accuracy, within 0.5 points zero-shot and within 5% control validation loss, at an average state precision near 5 bits. The comparison is at matched memory traffic per token, measured with a hardware profiler.
Two ablations isolate the key ingredient. Uniform INT4 state with Quamba's rotations tests whether non-uniformity matters at all. Magnitude-ordered allocation at the same byte budget tests whether the decay-gain ordering, rather than any mixed precision, does the work. A third ablation removes envelope-tracked scaling to measure what the borrowed zoom-in mechanism contributes.
Why not just…?
Why not just keep the state at 8 bits, or in FP16, and quantize everything else? At batch size one with a large model, weights dominate memory traffic and state precision barely matters. In that regime this objection wins. The proposal matters when many streams share one model: a fleet controller serving dozens of agents, batched decoding, or long contexts. There, per-stream state traffic grows with batch while weight traffic does not. We will report the batch size at which state bytes exceed weight bytes, and we restrict our speed claims to settings beyond it.
Why not just run an empirical per-channel sensitivity search? This is the real threat. A greedy search that quantizes channels one at a time and keeps the least harmful will likely find the same long-memory channels when calibration and test data match. The decay rule offers two things a search does not: it is nearly free to compute, and it rests on a quantity that moves predictably under input shift. The dropout experiments settle the question. Both allocations are calibrated on clean streams and evaluated with packet loss. If the empirical search remains as good, we report the decay rule as an explanation and a cheap proxy, not as a better allocator.
Why not just use stochastic rounding or error feedback on the state? Stochastic rounding removes bias but leaves variance, and the variance is still amplified by $1/(1-\bar a^2)$. It therefore complements gain-aware allocation rather than replacing it, and we include it both alone and combined. Classic error feedback needs a full-precision residual buffer, which restores the traffic we are trying to remove.
Why not just do quantization-aware training? Training could teach the model to tolerate 4-bit state. It costs a training run per model, and it is not available for the frontier checkpoints that post-training methods target. We run it once on a small model as an upper reference, to bound how much a post-training method could still gain.
Proposed Experiments
The language-modeling check uses WikiText-2 perplexity and the zero-shot suite of LAMBADA, HellaSwag, PIQA, ARC, and WinoGrande on Mamba 130M–2.8B, compared directly against Quamba's protocol.
The control experiments train Mamba-2 and diagonal-SSD models as one-step predictors on Flash STU's online linear dynamical systems and MuJoCo Ant tasks. They are then deployed as delay-compensating predictors in closed loop. Observations pass through a channel with Bernoulli dropout probability $1-\lambda$ and fixed delays of 2 to 20 steps. The same setup is applied to the connected-vehicle platoon simulation of Zhou et al., where the nominal controller is replaced by the SSM-based predictor.
The decisive measurement is the empirical critical arrival rate: the smallest $\lambda$ at which closed-loop tracking error stays bounded, plotted against average state bits for each allocation policy. The analogy to Sinopoli et al. predicts that uniform 4-bit state raises this threshold sharply. Gain-aware allocation should keep it near the FP16 value. Platoon tracking error is reported as RMSE, the metric used in that benchmark.
Hardware results report bytes moved per token and wall-clock decode latency on Orin Nano 8G across batch sizes.
The single result that would most strongly confirm the idea is a flat critical-$\lambda$ curve for gain-aware allocation at about 5 average bits, with magnitude-ordered allocation at the same budget degrading. The result that would most strongly refute it is the diagnostic stage finding no rank correlation between per-channel quantization damage and $1/(1-\bar a_i^2)$.
References
- SepMamba: State-Space Models for Speaker Separation Using Mamba
- LocalMamba: Visual State Space Model with Windowed Selective Scan
- Flash STU: Fast Spectral Transform Units
- Quamba: A Post-Training Quantization Recipe for Selective State Space Models
- SSDi8: Accurate and Efficient 8-bit Quantization for State Space Duality
- QMamba: Post-Training Quantization for Vision State Space Models
- On Structured State-Space Duality
- Progressive quantization in distributed average consensus
- Fast Convergence Rates of Distributed Subgradient Methods With Adaptive Quantization
- Innovation Compression for Communication-Efficient Distributed Optimization With Linear Convergence
- Kalman filtering with intermittent observations
- Input-to-State Safety with Input Delay in Longitudinal Vehicle Control
- Neural Vector Lyapunov–Razumikhin Certificates for Delayed Interconnected Systems