Heterogeneity-Aware Bit Allocation for SRAM-Resident Models

Tiny devices such as microcontrollers run neural networks fastest and cheapest when every weight and activation fits in the small memory on the chip itself (on-chip SRAM, typically 60 KB to a few MB). Fetching data from memory off the chip costs roughly 100–200× more energy than reading SRAM (TinyTL). To make a model fit, engineers compress it, mostly by storing different layers at different numbers of bits. They choose which layers keep more bits by measuring how much accuracy each layer loses on a single pooled test set. A deployed keyword spotter, however, does not serve a pooled test set. It serves one speaker, one room, or one machine at a time. Compression that costs 1% on average can cost far more for particular users, and averaging over the pooled set hides this.

We propose to treat each deployment environment (a speaker, a machine unit, a scene type) as a separate "study" in the sense of random-effects meta-analysis. Random-effects meta-analysis is the statistical method medicine uses to pool results from trials whose true effects differ from site to site. Concretely, for each layer and each candidate bit-width, we estimate two quantities: the mean accuracy loss across environments, and the between-environment spread of that loss ($\tau^2$), kept separate from the sampling noise of small per-environment test sets. We then choose bit-widths to stay within the SRAM budget while minimizing the loss a new, unseen environment would plausibly suffer. The empirical claim to test is that compressing a model to fit SRAM raises this spread, not only the mean loss, and that allocating bits with the spread in view protects the worst-served users at no extra memory.

Core Claim

Existing sensitivity-driven allocation methods assume one fixed test distribution. That holds whether they use Hessian curvature (APTQ, Neural Network Compression Framework, Adaptive Neural Network Quantization For Lightweight Speaker Verification), first-order Taylor scores (Distribution-Aware Adaptive Multi-Bit Quantization), or learned search (MetaMix, whose "meta" refers to a shared weight state across precisions, not to statistical pooling). In meta-analytic terms, all of them use a fixed-effect assumption that sets $\tau^2 = 0$.

The proposal replaces the per-layer sensitivity score with a random-effects prediction bound. For layer $l$ at bit-width $b$, let $d_{il}(b)$ be the accuracy drop in environment $i$, with sampling variance $\sigma_{il}^2$ from that environment's finite test set. We fit $d_{il} = \mu_l(b) + u_{il} + \epsilon_{il}$, where $u_{il}$ has variance $\tau_l^2(b)$ (true between-environment spread) and $\epsilon_{il}$ has variance $\sigma_{il}^2$. We estimate the between-environment variance by restricted maximum likelihood (REML), and score each choice by the upper bound of the drop expected in a new environment:

$$ S_l(b) = \hat\mu_l(b) + z_{\alpha}\sqrt{\hat\tau_l^2(b) + \widehat{\mathrm{Var}}(\hat\mu_l(b))}, $$

where $z_\alpha$ is the normal quantile for the chosen protection level, for example the 90th percentile. Allocation then solves a knapsack problem: minimize $\sum_l S_l(b_l)$ subject to $\sum_l \mathrm{mem}_l(b_l) \le M_{\text{SRAM}}$, where $\mathrm{mem}_l$ is the layer's weight storage. Peak activation memory is checked separately with the in-place scheduling of MCUNet.

The meta-analytic machinery earns its place over simply ranking environments for one reason. Edge environments typically come with very few labelled examples each, such as a few dozen utterances per speaker. With so little data per environment, raw per-environment drops mix true heterogeneity with test-set noise, and the random-effects decomposition separates the two.

Research Program

The first stage is diagnostic. It asks whether SRAM-driven compression inflates heterogeneity at all. The clinical literature in the file shows that pooled estimates which ignore $\tau^2$ produce overconfident conclusions under cross-site variation (Vitamin D supplementation for tuberculosis prevention, Risk factors of epidural hematoma). That evidence motivates the question but does not answer it for neural networks. We will compress MLPerf Tiny models step by step, using uniform post-training quantization and then magnitude pruning in the style of Han et al.. At each step we fit the random-effects model to per-environment accuracy and track $\hat\mu$ and $\hat\tau^2$ separately. The hypothesis, raised in the file's Gap 3 but never tested, is that pruning removes capacity that only some environments rely on. If so, $\hat\tau^2$ should grow faster than $\hat\mu$ as the memory budget shrinks.

The second stage builds the allocator described above. It computes per-layer, per-bit random-effects scores on a calibration split of environments and solves the knapsack exactly with dynamic programming over discretized memory. For streaming speech recognition we add the invocation-frequency weighting of Li et al.. Their finding was that the Joiner, the small component that combines audio and text predictions, accounts for 5–9% of parameters but 48–73% of power. Our extension asks whether the Joiner's compression error is also heterogeneous across speakers, and whether moving it into SRAM at low precision is safe for all speakers or only on average.

The third stage tests calibration on environments the allocator never saw. Environments are split into those used to fit the model and those held out. The question is whether the random-effects prediction interval actually covers the accuracies of held-out environments at its nominal rate.

Baselines

The naive reference point is uniform INT8 post-training quantization, which MLPerf Tiny reports as the dominant submission format. Where INT8 overflows the SRAM budget, the reference is uniform INT4 at the same budget. This tells us how much any allocation method buys. Beating it is necessary but not interesting.

The strongest existing competitor for choosing bits per layer is Hessian-trace mixed-precision allocation computed on the pooled calibration set, as in APTQ and Neural Network Compression Framework. We also run MetaMix as the strongest learned search. MetaMix's reported ImageNet-1K result is 72.60% Top-1 for MobileNet-v2 at a 4-bit target, versus 71.56% for PROFIT. For the per-speaker keyword-spotting and per-machine anomaly-detection evaluations used here, no reported result was found on Lacuna. On Visual Wake Words, the closest memory-matched reference is RaScaNet at about 91.6% accuracy in under 60 KB. RaScaNet's architecture serves as a backbone for a separate comparison, not as a competing allocator.

Every allocator receives the same pretrained backbone, the same candidate bit-widths {2, 3, 4, 8}, the same calibration environments and samples, the same SRAM budget, and the same post-allocation fine-tuning budget measured in GPU hours. The proposal beats these baselines if it raises the 10th-percentile accuracy over held-out environments, keeps mean accuracy within 0.5 points of the best baseline at equal SRAM, and repeats this across at least three memory budgets.

Two ablations isolate the key ingredient. The fixed-effect ablation sets $\hat\tau^2 = 0$ in $S_l$, which reduces the score to a precision-weighted mean drop. If it matches the full method, heterogeneity modelling adds nothing. The empirical-quantile ablation replaces the random-effects bound with the raw 90th percentile of per-environment drops. It isolates whether separating $\tau^2$ from sampling noise matters, or whether simply looking at the tail is enough.

Why not just…?

Why not just average accuracy over all environments, weighted by sample size? This is the federated-averaging instinct and equals the fixed-effect model. When true effects vary across sites, fixed-effect pooling lets large sites dominate and understates uncertainty. That is the central lesson of the random-effects literature (Thyroxine changes in COVID-19). The fixed-effect ablation tests this directly. If $\hat\tau^2$ turns out near zero for these tasks, it will win, and we would report that compression-induced heterogeneity is not a practical problem.

Why not just optimize for the worst environment, as in Group DRO? Minimizing the worst-case loss (Robust Learning of A Group DRO Neuron, Factored DRO) is a real alternative. With few test examples per environment, though, the empirically worst environment is often just the noisiest one, so the allocator ends up chasing test-set noise. We run a worst-environment allocator as an additional baseline under the same budget. The random-effects bound should match its tail accuracy while losing less mean accuracy. If DRO matches both, the simpler method wins.

Why not just take the empirical 10th percentile across environments? This is the most serious threat. With many environments and large per-environment test sets, the empirical quantile and the random-effects bound converge. The empirical-quantile ablation settles the question. We will vary per-environment test size by subsampling utterances per speaker. Our prediction is that the random-effects method's advantage is largest at small sizes and disappears at large ones. If no advantage appears even at small sizes, the meta-analytic layer is unnecessary.

Proposed Experiments

Keyword spotting on Speech Commands v2 from MLPerf Tiny is the primary testbed, with speakers as environments. It offers hundreds of environments with few utterances each, which is exactly the regime random-effects pooling is built for. Anomaly detection on ToyADMOS, with individual machine units as environments, tests the regime of few environments. Here the file notes that estimating $\tau^2$ becomes unstable when there are fewer than five studies. We therefore compare REML against the Paule-Mandel estimator and report where the method breaks. Streaming speech recognition on LibriSpeech, grouped by speaker, tests the Joiner extension. Here the reference result is the 47% energy reduction and 29% real-time-factor improvement reported by Li et al.. Our measure is whether low-precision Joiner residency increases between-speaker spread in word error rate. Memory budgets follow the STM32H743 profile used by MCUNet (512 KB SRAM) and tighter 256 KB and 128 KB cuts.

Four results decide the proposal. The first is whether $\hat\tau^2$ rises measurably as SRAM budgets shrink; if it does not, the premise fails. The second is whether the random-effects prediction interval covers held-out environment accuracies at close to its nominal rate; if not, the score is miscalibrated and should not guide allocation. The third is whether, at equal SRAM, the allocator improves held-out tail accuracy over Hessian-pooled allocation and MetaMix without the mean falling by more than 0.5 points. The fourth is whether this advantage over the empirical-quantile ablation shrinks as per-environment data grows, as the statistical argument predicts. We report every comparison across seeds as a random-effects pooled effect with an interval, so the paper's own claims meet the standard it proposes.