Speech features computed on a phone or wearable can protect privacy, because they hold far less than the raw waveform. Recent systems turn audio into a stream of discrete "speech units", phoneme-like tokens drawn from a small vocabulary, and send only those tokens off the device. This scheme breaks in crowded rooms. When the background is multi-talker chatter ("babble"), the extractor transcribes the bystanders' phonemes into the unit stream. That hurts accuracy, and it also leaks. Content spoken by people who never agreed to be recorded now leaves the device inside a supposedly privacy-preserving code. The fix that follows is to adapt the extractor to the babble of each user's own surroundings, on the device, without uploading any raw audio. The obstacle is memory. Backpropagation must keep every intermediate layer output (the "activations") in RAM, and on edge hardware those saved activations are the largest memory cost.
We propose storing those saved activations and the backward-pass gradients at 4 bits using stochastic rounding. Stochastic rounding rounds a number up or down at random, with probabilities chosen so that the rounded value equals the original on average. Its known weakness is that its noise must be averaged away by large mini-batches, which an edge device cannot afford. Our central observation is that audio supplies the averaging for free. A single utterance holds hundreds of frames, and the rounding noise is drawn independently at each frame. So in the gradient of any shared weight, the number of frames plays the role that batch size plays in existing analyses. A device can adapt on one utterance at a time (batch size 1) and still have rounding variance suppressed. The rounding errors of ordinary deterministic rounding are fixed functions of the signal, so they do not average out this way.
Core Claim
Training with Fewer Bits proves that stochastic-rounding variance in per-sample activations and gradients decays as $1/b$ in the mini-batch size $b$. Its practical rule is that one bit of precision costs roughly a doubling of batch size. It evaluates only text and image models. The paper also shows that noise from quantizing the weights themselves does not decay with batch size. The file frames this as a dilemma: edge devices need $b=1$, while stochastic rounding needs large $b$.
We argue this dilemma largely disappears for streaming speech. The claim holds only if two things are kept separate. The first is the correlation of the signal across time, which is high in audio. The second is the correlation of the rounding errors, which are independent draws given the values. Consider a weight matrix applied at every frame $t=1,\dots,T$. Its gradient is a sum of per-frame terms $a_t\delta_t^\top$, where $a_t$ is the stored input activation and $\delta_t$ is the back-propagated error. Stochastic rounding of $a_t$ to a grid of spacing $\Delta$ adds an error $e_t$ with zero mean and per-entry variance at most $\Delta^2/4$. The error in the frame-averaged gradient therefore satisfies
$$ \mathrm{Var}\Big[\tfrac{1}{T}\textstyle\sum_{t=1}^{T} e_t\,\delta_t^\top\Big] \;\le\; \frac{\Delta^2}{4\,b\,T}\,\overline{\delta^2}, $$
where $\overline{\delta^2}$ is the mean squared back-propagated error. The $1/(bT)$ form follows because the variances of independent zero-mean terms add. Deterministic round-to-nearest instead contributes a bias that is a fixed function of $a_t$. Because babble-corrupted speech frames are strongly correlated, that bias adds up coherently rather than cancelling. The bound concerns absolute error. The relative benefit also requires the true per-frame gradient terms to add up coherently rather than cancel, so we measure this directly instead of assuming it.
This claim is kept deliberately narrow. Stochastic rounding is applied where the memory is, namely the saved activations and the gradients passed backward through a frozen, quantized encoder. The small set of trainable adapter weights stays in higher precision. The reason is practical. For a few thousand adapter parameters, a full-precision copy costs almost nothing, so arguing for 4-bit weights there would be solving a problem that does not exist.
Why this intersection, and what is already known
Three papers set up the problem. How Generative Spoken Language Modeling Encodes Noisy Speech shows that a HuBERT unit encoder fed babble-corrupted speech encodes the background talkers' phonemes. At 0 dB in heavy babble (H-BAB, from DEMAND), the unit error rate reaches 54.1% and the word error rate of resynthesized speech reaches 75.8%. Naturalness scores stay high throughout, which the authors call the "naturalness trap". They analyse this failure but propose no remedy and use no edge constraints. We reinterpret their finding as a bystander-privacy leak and treat it as the thing to be reduced.
On-device Streaming Discrete Speech Units makes a WavLM-based unit extractor streamable with layer pruning, local attention windows, and a trainable linear unit predictor. It reports 6.3 WER on clean LibriSpeech at 0.96 TFLOPs per minute of audio, versus 5.0 WER for the full model, and a bandwidth reduction from 512 to 0.6 kbps. It is the system we adapt. Its evaluation is on clean speech only, with no quantized training and no adversarial privacy tests.
Training with Fewer Bits supplies the variance analysis we extend from batch to frames. The broader stochastic-rounding literature supplies the method and its limits. Deep Learning with Limited Numerical Precision establishes that stochastic rounding is unbiased. A Statistical Framework for Low-bitwidth Training locates the failure of low-bit training in gradient variance rather than bias. Accurate Neural Training with 4-bit Matrix Multiplications argues that unbiasedness matters more than mean-squared error for SGD. Together these predict that 4-bit stochastic rounding should behave like noisy but correct training, and 4-bit deterministic rounding like biased training.
On Stochastic Rounding with Few Random Bits constrains the design. It shows that random-bit generation has a hardware cost, so we test random-bit budgets explicitly rather than assume free entropy.
Two lines constrain how adaptation can be done. DOTN adapts speech enhancement to new noise with optimal transport and a Wasserstein discriminator. It notes that discriminators rewarding "speech-like" output end up rewarding babble, and it needs server-class compute. Both points rule out adversarial objectives for our on-device loop. The Model Hears You shows that downstream guardrails do not stop identity inference from audio. That supports removing information before transmission rather than filtering it afterward. The compute-and-destroy practice from Haider et al. sets our data-handling rule: captured audio lives only in volatile memory for one adaptation step, then is deleted.
Research Program
The system starts from the streaming unit extractor of Choi et al. with its encoder frozen and stored in INT8. The trainable part is the linear unit predictor plus low-rank adapters (small added matrices) in the top $k$ encoder layers.
The adaptation signal must work without labels and without clean recordings of the user's environment, and it must not use a discriminator. A small bank of public clean utterances ships with the device, a few minutes of LibriSpeech. At each step the device captures a short window of ambient audio and overlays a bank utterance at a random signal-to-noise ratio. It then trains the adapted extractor so that its unit predictions on the mixture match the units that the frozen extractor produces on the clean bank utterance. This is a teacher–student consistency loss, cross-entropy against the clean-speech units. The ambient buffer is then discarded.
This objective directly penalizes the naturalness trap. Any unit that tracks a bystander rather than the bank utterance counts as an error. It also avoids the discriminator failure documented for DOTN. The user's own speech may appear in the captured window. That is harmless, because the target is the overlaid utterance and the user's speech acts as extra interference.
In the backward pass, the stored activations for the $k$ adapted layers and the back-propagated gradients are kept at 4 bits (and swept down to 2) using stochastic rounding. The adapter weights and optimizer state stay in FP16 or FP32. The work has three parts.
The first part is analysis. We measure empirically how gradient error variance depends on the number of frames $T$ and on bit-width. We also measure how it depends on the coherence of per-frame gradient terms, since that coherence decides whether the $1/T$ bound translates into a useful signal-to-noise ratio.
The second part is the accuracy and privacy outcome. Adaptation should lower the target unit error rate under babble. It should also lower bystander leakage, meaning how much of the background talkers' speech can be recovered from the transmitted units.
The third part is cost. We measure peak memory, latency, and energy per adaptation step on real hardware, alongside the random-bit budget.
Baselines
The naive reference point is the unadapted streaming extractor of Choi et al., run on our babble mixtures. It has no reported result under babble on Lacuna; its only reported figure is 6.3 WER on clean LibriSpeech. We therefore measure its babble performance ourselves, and every improvement is stated relative to it. As an external anchor for how badly unit encoders fail, the full HuBERT pipeline of Park et al. reports 54.1% unit error rate and 75.8% resynthesized WER at 0 dB in heavy babble. We rerun our extractor on the same DEMAND condition so the two are comparable.
The strongest existing competitor is offline multi-condition training. Here the same extractor and adapters are trained once, on the server, with babble augmentation drawn from many environments and talker counts, and then shipped frozen. The adaptation literature on Lacuna reports no result for this on unit extractors. It is the strongest competitor because it costs nothing on the device and may already generalize to unseen babble. We give it the same trainable parameters, the same bank utterances as clean sources, a babble pool at least as large as all local environments combined (excluding the test environments), and a hyperparameter search at least as large as ours. On-device adaptation beats it only if, averaged over held-out environments, it reduces target unit error rate by a margin larger than the across-seed spread. It must also either improve or leave unchanged bystander leakage. Starting on-device adaptation from the offline-augmented model is also tested, since the two methods are complementary.
The ablations isolate the key ingredient. Adaptation with FP32 stored activations is the accuracy ceiling. Adaptation with gradient checkpointing (recomputing activations during the backward pass instead of storing them) matches the ceiling's accuracy at extra compute. Adaptation with 4-bit round-to-nearest stored activations isolates rounding bias. Head-only adaptation trains just the linear unit predictor, needs no stored encoder activations, and isolates whether backprop into the encoder is needed at all. Finally, stochastic rounding with a variable number of frames per step, from 16 up to the full utterance, isolates the frames-as-batch effect.
All variants use the same seeds, the same number of adaptation steps, the same ambient audio per device, and identical learning-rate searches. Comparisons at matched peak memory and at matched energy are both reported.
Why not just…?
Why not just train offline with babble augmentation and skip on-device learning? This is the most serious threat. Babble statistics vary strongly with the number of talkers: Krishnamurthy et al. show that small crowds produce sparse, speaker-like bursts and large crowds approach stationary noise. That argues for local specialization, but it does not prove it. The multi-condition baseline above settles the question directly. If offline augmentation matches adaptation on held-out environments, we will report that on-device adaptation is unnecessary for unit extractors. The stochastic-rounding analysis would then stand as a result about when frame averaging substitutes for batch size, rather than as a deployment recommendation.
Why not just use deterministic 8-bit or 4-bit rounding, as mobile toolchains already support? The argument for stochastic rounding rests on bias. Round-to-nearest errors on correlated audio frames are correlated and add up rather than cancel (Gupta et al., Chen et al.). The round-to-nearest ablation at 8, 4, 3 and 2 bits tests this. We expect parity at 8 bits and a widening gap at 4 bits and below. If round-to-nearest matches stochastic rounding at 4 bits, the memory saving still stands, but the case for stochastic rounding does not.
Why not just recompute activations (gradient checkpointing) or adapt only the output head? Checkpointing removes the memory problem without any precision loss, but it adds roughly a forward pass per step. Adapting only the output head removes both memory and compute costs. We compare both at matched energy per step. Head-only adaptation is the more likely threat. If babble errors arise in the encoder's representations rather than in the final mapping to units, head-only adaptation should plateau early. If it does not plateau, the simpler method wins and we will say so.
Why not just upload audio and adapt on a server, or run an enhancement front-end? Uploading babble means uploading bystanders' speech, which violates the compute-and-destroy constraint that motivates this line of work. Enhancement front-ends that are strong on babble, such as DOTN, depend on discriminator games and server compute. Classical filters fail when noise shares speech's spectral structure (Raj et al.).
Proposed Experiments
We construct evaluation mixtures from LibriSpeech target utterances. The interference comes from two sources: DEMAND babble at the L-BAB, M-BAB and H-BAB conditions used by Park et al., and synthetic babble from 1 to 8 other LibriSpeech speakers. For the synthetic babble, the bystander transcripts are known, so leakage can be measured exactly.
Each simulated device is assigned one held-out babble environment. It receives a few minutes of ambient audio for adaptation and is evaluated on different target speakers in the same environment. Target quality is measured three ways: unit error rate against the units from clean speech, WER of a unit-to-text recognizer, and resynthesized WER, reported so that it can be compared with Park et al.
Privacy is measured with two attackers trained on unadapted units. The first is a bystander-transcription attacker, a unit-to-text model trained to recover the background speech. Its WER measures leakage, and higher is better. The second is a speaker-verification attacker, whose equal error rate on target-speaker identity is also better when higher. Utility on clean speech is checked on LibriSpeech against the reported 6.3 WER. A regression larger than about 0.5 absolute WER counts as a failure.
Cost is measured on a physical device, a phone-class ARM board: peak training memory, latency, and energy per adaptation step, plus the random bits consumed. We sweep random-bit budgets following On Stochastic Rounding with Few Random Bits.
Three checks decide whether the proposal succeeds. First, gradient error variance under stochastic rounding, measured against the FP32 gradient on a fixed utterance, should fall close to $1/T$ as frames are added, and should not fall for round-to-nearest. Second, 4-bit stochastic-rounding adaptation should come within across-seed noise of FP32 adaptation on target unit error rate at batch size 1, while cutting stored-activation memory roughly fourfold relative to FP16. Third, it should beat both the unadapted extractor and the offline multi-condition baseline on H-BAB at 0 to 5 dB, and should raise the bystander attacker's WER.
If the first check fails, the frames-as-batch argument is wrong for real audio. If the third fails against offline augmentation, the deployment case is wrong. Both are informative outcomes and would be reported as such.
References
- Training with Fewer Bits: Unlocking Edge LLMs Training with Stochastic Rounding
- How Generative Spoken Language Modeling Encodes Noisy Speech: Investigation from Phonetics to Syntactics
- On-device Streaming Discrete Speech Units
- Deep Learning with Limited Numerical Precision
- A Statistical Framework for Low-bitwidth Training of Deep Neural Networks
- Accurate Neural Training with 4-bit Matrix Multiplications at Standard Formats
- On Stochastic Rounding with Few Random Bits
- Unsupervised Noise Adaptive Speech Enhancement by Discriminator-Constrained Optimal Transport
- The Model Hears You: Audio Language Model Deployments Should Consider the Principle of Least Privilege
- A System for Real-Time Privacy Preserving Data Collection for Ambient Assisted Living
- Babble speech: acoustic and perceptual variability
- Regularized non-negative matrix factorization with temporal dependencies for speech denoising