Speaker verification models that decide whether a voice matches an enrolled user have recently been shrunk below one megabyte by binarizing their weights, meaning each weight is stored as one of two learned values. That is small enough, on paper, to sit in the fast memory built into a chip rather than in external DRAM. On such hardware, fetching data from DRAM costs roughly a hundred times more energy than reading on-chip memory. The papers reporting these sub-megabyte models count only weights, though. The intermediate feature maps a convolutional network produces while it processes a spectrogram (a time-by-frequency picture of the audio) are kept in 32-bit floats, and they grow linearly with utterance length. For a ResNet on a few seconds of speech, those feature maps can outweigh the binarized weights several times over. On a real device they would spill to DRAM, which is exactly the cost the compression was meant to remove.
This note proposes to make the whole speaker embedding computation fit on-chip, weights and working memory together, under one fixed byte budget. The central observation is that the statistics pooling layer, which turns a variable-length sequence of frame features into one fixed-size speaker vector, only needs running sums of each feature and its square. The network can therefore process audio in short time chunks. Each convolution keeps only a small cache of recent frames, and the resulting embedding is the same as the one computed over the full utterance. Working memory then becomes a constant that does not depend on how long the person speaks. The memory this frees can be spent on higher-precision or wider weights. The research question is how a fixed on-chip budget should be divided between weight precision, network width and activation precision once activations are counted honestly, and how much equal error rate that division buys.
Core Claim
We claim three things. First, for existing sub-megabyte speaker verification models, activation memory rather than weight memory is the binding on-chip constraint at realistic utterance lengths. Second, chunked streaming with running-statistics pooling bounds that memory at a small constant while leaving the embedding mathematically unchanged, apart from quantization of the cached frames. Third, when the full byte budget is allocated jointly, a model constrained to 1 MB of total on-chip memory can beat the equal error rate of today's best "under 1 MB" model, whose reported size does not include activations. We also expect a usable model at 512 KB and 256 KB total.
The equal error rate is the operating point where false accepts equal false rejects, and lower is better. The new element is not quantization or streaming convolution taken separately. It is treating the speaker network's pooling structure as a license for exact, length-independent execution, and then re-deriving the compression-accuracy trade-off under a total-memory constraint instead of a weights-only one.
Where This Sits in Prior Work
Three quantization papers on speaker verification supply the compression methods and motivate the gap. Extremely Low Bit Quantization for Mobile Speaker Verification Systems Under 1MB Memory binarizes ResNet34 with a learned center and spread per layer, reaching 0.97 MB and 1.72% EER on Vox1-O, against 0.89% for the full-precision model. Towards Lightweight Speaker Verification via Adaptive Neural Network Quantization adds k-means quantization-aware training and assigns bits per layer using the Hessian trace as a sensitivity signal, reporting 3.14% EER on Vox1-H at 1 bit. Model Compression for DNN-based Speaker Verification Using Weight Quantization finds that ResNet34 tolerates 4-bit weights (1.10% to 1.22% EER, 3.52 MB) far better than ECAPA-TDNN (1.07% to 1.56%). It also finds that quantized models lean more on background scene cues. All three leave activations in floating point and never measure memory placement. We reuse their quantizers and their Hessian sensitivity signal, and we change the constraint they optimize against.
Boundary-Conscious Pruning shows that compression damages the hard trials near the decision boundary most. On ECAPA-TDNN at 50% channel pruning, it reports a hard-set EER of 4.96% versus 6.92% for one-shot pruning. We take from it the lesson that hard-trial EER must be reported separately, and we include its channel pruning as a way to buy width reductions.
The memory-side methods come from speech recognition and vision. Breaking Down Power Barriers in On-Device Streaming ASR and Memory-efficient Speech Recognition on Smart Devices show that DRAM traffic, not parameter count, drives energy. The latter sizes a recurrent encoder for a 512 KB buffer and cuts off-chip traffic 4.5×. Both concern speech recognition transducers, whose per-frame outputs make streaming natural. Neither considers an utterance-level embedding network. RaScaNet attacks the same peak-activation problem for images by scanning row strips and carrying context in a recurrent cell, keeping total memory under 60 KB. Our setting differs in a useful way. Because statistics pooling is a sum over time, no learned recurrent state is needed to carry context, so streaming is exact rather than a new architecture that must be retrained and may lose accuracy.
Research Program
The first step is measurement. We will build a static memory planner for ResNet34, a half-width ResNet, and ECAPA-TDNN on 80-dimensional filterbanks. For each utterance length from 2 to 20 seconds, it will compute peak live bytes for weights, activations, pooling buffers and scratch space, under conventional whole-utterance execution. My rough count for a 32-channel-base ResNet34 puts first-stage float activations in the megabytes for a few seconds of audio. That is an estimate to be verified here, not a reported figure. If the planner shows that activations are not dominant, the premise fails and the note's direction changes toward pure weight allocation.
The second step is exact streaming. Each time-axis convolution keeps a ring buffer of its last $k-1$ input frames, where $k$ is its temporal kernel extent after accounting for stride, so each chunk's output equals the corresponding slice of the full-sequence output. Pooling accumulates $\sum_t h_t$ and $\sum_t h_t^2$ over frame features $h_t$ in higher precision. Mean and standard deviation are formed at the end. Two components break exactness, and we will treat them explicitly.
The first is utterance-level mean and variance normalization of the input spectrogram. VoxCeleb found this normalization worth nearly 10% identification accuracy. We will compare a running-mean estimate against a two-pass variant that first computes normalization statistics and then streams.
The second is ECAPA's attentive pooling, whose attention scores take the global mean as input. If the scores depend only on the current frame, attentive pooling can be computed in one pass with a running maximum and running normalizer, the same trick used in memory-efficient attention. We will test that local-score variant against the original.
The third step is budgeted allocation. Let $P_l$ be the parameter count of layer $l$ and $H_l$ the number of cached activation values in its ring buffer. Let $b^w_l$ and $b^a_l$ be the weight and activation bit-widths, and let $O$ be the fixed overhead for pooling accumulators and input chunk. We require
$$\frac{1}{8}\sum_l \left(P_l\, b^w_l + H_l\, b^a_l\right) + O \le S,$$
where $S$ is the on-chip budget in bytes. All layers' ring buffers are live simultaneously while streaming, which is why the constraint is a sum rather than a maximum over layers. The allocation minimizes a Hessian-trace sensitivity proxy, following the adaptive quantization paper. Width reductions come from boundary-conscious channel pruning. We will search over $S \in \{1\,\text{MB}, 512\,\text{KB}, 256\,\text{KB}\}$ and fine-tune each chosen configuration with quantization-aware training for both weights and activations. The hypothesis is that under streaming, the optimal configuration puts nearly all bits into weights, with activations at 8 or fewer bits costing only tens of kilobytes, whereas without streaming the budget is swallowed by activations.
Baselines
The naive reference point is the full-precision ResNet34 run in the usual whole-utterance way. It reaches 0.89% EER on Vox1-O as reported by the 1 MB quantization paper, and 1.10% in the weight-quantization paper's own training. We will report its total memory as measured by our planner, which shows how far off on-chip it sits.
The strongest existing method on this question is the adaptive binarized ResNet34 ("b-vector") from Extremely Low Bit Quantization for Mobile Speaker Verification Systems Under 1MB Memory, at 0.97 MB of weights and 1.72% EER on Vox1-O. The Hessian mixed-precision 1-bit ResNet34 at 3.14% EER on Vox1-H from Towards Lightweight Speaker Verification via Adaptive Neural Network Quantization is the reference for hard trials. We will reimplement both and train them on VoxCeleb2 with the same augmentation (MUSAN noise and reverberation), epoch count and quantization-aware fine-tuning budget as our models. We will then measure their true total footprint with the same planner, with activations left in float as their authors did and also quantized to 8 bits, so the comparison is at equal total bytes.
We also include 4-bit uniform ResNet34 from Model Compression for DNN-based Speaker Verification Using Weight Quantization (3.52 MB, 1.22% EER on Vox1-O), as a point well above the budget that anchors the accuracy ceiling. For ECAPA-TDNN we include boundary-conscious pruning at 50% (hard-set EER 4.96%).
Beating the strongest baseline means one of two things. The first is lower Vox1-O and Vox1-H EER than our reimplemented b-vector when both are held to 1 MB total on-chip memory for utterances up to 20 seconds. The second is matching the reported 1.72% at a strictly smaller total budget.
The key ablation removes streaming while keeping joint allocation, activation quantization and training fixed. This isolates whether the gain comes from length-independent memory or merely from counting activations. A second ablation keeps streaming but allocates bits by the weights-only rule the prior papers use.
Why not just…?
Why not just crop utterances to two seconds and average the embeddings of independent crops? This also bounds activation memory and is a real threat. Its weakness is that each crop's pooling statistics are noisy, and short segments are exactly where speaker verification degrades. Averaging normalized embeddings is also not equivalent to pooling statistics over the whole utterance. The experiment settles it directly: at equal memory, compare EER of crop-averaging against exact streaming on full-length Vox1-O, E and H trials, and on a truncated-duration sweep. If crop-averaging matches within noise, the exactness argument has no practical value, and we would report that honestly.
Why not just quantize activations to 8 bits and run the whole utterance? This quarters activation memory but leaves it proportional to utterance length, so any fixed on-chip budget is exceeded past some duration. The planner will report that break-even length for each architecture. The ablation that removes streaming measures how much weight precision must be sacrificed to keep activations in budget for 20-second inputs.
Why not just train a small architecture such as ECAPA-TDNNLite from scratch? The file's evidence is that the 0.97 MB binarized ResNet34 beats ECAPA-TDNNLite while using 24% less memory. Adaptive 1-bit ResNet34 is 35% better on Vox1-H. That comparison, however, counted weights only. One-dimensional TDNN activations have no frequency axis and may be far smaller than a 2D ResNet's, so under a total-memory budget the ranking could flip. We therefore run ECAPA-TDNN and a compact TDNN through the same planner and streaming treatment. We accept that the outcome may favor a TDNN, although ECAPA's weight distribution, which is concentrated sharply around zero, is known to quantize badly.
Why not just leave the model in DRAM? Phones have plenty of DRAM. The motivation is always-on verification on wearables and smart-home hardware, where Breaking Down Power Barriers in On-Device Streaming ASR reports DRAM access at nearly 100× the energy of on-chip memory. On that class of hardware, keeping DRAM powered down is the point. We will report estimated energy per verification from counted memory accesses to make this concrete. We concede that on phones the benefit may be small.
Proposed Experiments
We train on VoxCeleb2 and evaluate on Vox1-O, Vox1-E and Vox1-H, reporting EER and minimum detection cost. Hard-set EER follows the boundary-conscious pruning definition. CN-Celeb serves as a cross-domain check, since the weight-quantization paper found that quantization both regularizes cross-lingual transfer and raises scene bias. For each model we report peak on-chip bytes as a function of utterance length and counted DRAM accesses per verification. Latency is measured on a microcontroller-class target with 512 KB SRAM, alongside the planner's simulation.
The first decisive check is exactness. Streamed embeddings in float must match whole-utterance embeddings to numerical precision, and after quantization their cosine similarity to the float embedding should be indistinguishable from that of the non-streamed quantized model. Any EER gap between streamed and non-streamed versions of the same weights isolates the cost of the online normalization and local-score attention approximations.
The second check is the budget curve: EER at 1 MB, 512 KB and 256 KB total, against our reimplemented baselines forced to the same totals. The third check tests the allocation hypothesis. If the searched allocations under streaming do not shift bits from activations to weights relative to the non-streamed search, the mechanism we propose is not what is producing the gains.
Finally, because the weight-quantization paper found that compressed speaker models lean on scene cues, we evaluate the smallest budgets under MUSAN-noise and reverberation mismatch. This tells us whether memory savings bought on clean trials survive a change of acoustic environment.
References
- Extremely Low Bit Quantization for Mobile Speaker Verification Systems Under 1MB Memory
- Towards Lightweight Speaker Verification via Adaptive Neural Network Quantization
- Model Compression for DNN-based Speaker Verification Using Weight Quantization
- Boundary-Conscious Pruning: Hard Set-Aware Model Compression for Efficient Speaker Recognition
- Breaking Down Power Barriers in On-Device Streaming ASR: Insights and Solutions
- Memory-Efficient Speech Recognition on Smart Devices
- RaScaNet: Learning Tiny Models by Raster-Scanning Images
- VoxCeleb: A Large-Scale Speaker Identification Dataset