Least-Privilege Distillation for Microcontroller Speech Front-Ends

A voice device that sends raw audio to the cloud also sends the speaker's identity, emotional state, and a recording someone could replay or transcribe. To understand a command like "turn off the kitchen lights," the cloud needs none of that. Current on-device defenses force a choice. One option is a front-end small enough for a microcontroller that protects only against transcription. The other is a large "disentangling" network that removes identity cues but needs tens of millions of parameters. We propose to build the privacy filter by what we teach a tiny model, rather than by adding a filter on top of it.

The idea uses knowledge distillation, in which a small "student" network learns to reproduce the internal outputs of a large "teacher." We distill a sub-megabyte student from a deep layer of a large speech model. Before the teacher's representation is used as the training target, we remove every direction in it that predicts speaker identity or emotion. The student has little capacity, and its only objective is to reproduce this cleaned target, so it has no reason to spend that capacity encoding voiceprints. We test whether this "least-privilege" target gives a better privacy–utility–cost trade-off than the obvious alternatives. Those alternatives are training the tiny model on task labels alone, adversarial identity removal, and discrete speech units. The comparison uses spoken-intent benchmarks under realistic attackers.

Core Claim

A tiny student distilled from an identity-scrubbed deep-layer teacher target leaks less speaker and emotion information than an equally sized student trained on task labels or with an adversarial speaker penalty, at matched intent accuracy. It does this within a microcontroller memory budget. The claim has three parts, and each can fail on its own. The first is that scrubbing works at the teacher. The second is that the scrubbing survives transfer to a small student instead of being re-learned from the raw audio. The third is that distillation, rather than task supervision alone, is what gives the student its utility.

The claim builds on an observation from DAISY, which attaches early-exit branches to HuBERT and finds that speaker characteristics are resolved in shallow layers while phonetic and semantic content requires deep layers. DAISY uses this to save compute, and exiting early preserves identity. We use it in the opposite direction. The teacher target is taken from a deep layer, where identity is already weaker, and the remaining identity directions are then projected out. Formally, let $h_L(x)$ be the teacher's layer-$L$ representation of utterance $x$. Let $P$ be a projection that removes the subspace spanned by linear speaker and emotion probes, estimated by repeatedly fitting such probes on $h_L$ and removing their directions until they reach chance. The student $f_\theta$ is trained with

$$ \mathcal{L}(\theta) = \lVert f_\theta(x) - P\,h_L(x) \rVert_2^2 + \lambda\, \mathcal{L}_{\text{task}}(g(f_\theta(x)), y), $$

where $g$ is a small intent head, $y$ is the intent label, and $\lambda$ weights the task term. The projection removes only linear leakage. The hypothesis that must be tested is that a capacity-limited student cannot afford to rebuild the nonlinear residue, because it has no incentive to.

Relation to Prior Work

The closest system is SILENCE. It runs a tiny learned frame mask on an STM32H7 microcontroller (394.9 KB, 912 ms per 4 s clip) and zeroes most frames. This drives a malicious recognizer above 80% word error rate while keeping 81.2% intent accuracy on SLURP. SILENCE supplies our deployment target and our strongest baseline. It differs from our proposal in two ways. First, it defends only against transcription: the frames it keeps are raw audio, which still carries voiceprint and emotion. Second, it does not distill from a large model. On-device Streaming Discrete Speech Units prunes WavLM-Large into a streaming unit extractor (50% fewer FLOPs, 6.5% relative CER increase). It asserts that the units discard speaker identity but never measures this against a speaker-verification attacker. That makes it both a competitor and an untested privacy claim we will measure.

Two distillation papers supply the method and also warn against using it naively. FRILL distills a ResNet50 TRILL teacher into MobileNetV3 students with an MSE embedding loss (32× faster, about 2% average accuracy drop). We reuse its recipe and its model-size search, but FRILL is built to preserve speaker and emotion, so it is our "unscrubbed" ablation. Adaptive Knowledge Distillation for Device-Directed Speech Detection shows that distilling across a 16× size gap (a 79M-parameter Conformer to a 5M-parameter student) needs extra alignment terms. This constrains us: our gap is larger still, so we budget for an intermediate-size teacher-assistant step if direct transfer stalls.

Swing Distillation shows that distillation targets carry private information from teacher to student, and it defends against this with a text clue-word dictionary. That dictionary has no acoustic analogue. We take its lesson, that the target is the leak, and replace the dictionary with a learned identity subspace. InfoDecom scrubs features sent from a client to a server in split inference for images. Privacy-preserving Voice Analysis via Disentangled Representations and Privacy-Preserving Siamese Feature Extraction remove attributes with networks trained end to end. All three put the privacy mechanism in the deployed model. We move it into the training target, so the deployed model carries no filtering overhead. The motivation comes from The Model Hears You, which shows audio language models identifying public figures and inferring emotion from voice, and shows that software guardrails are easily bypassed. It argues for least privilege but offers no front-end that enforces it. Our student is that front-end.

Research Program

The teacher is WavLM-Large or HuBERT fine-tuned for intent on SLURP. We sweep the target layer $L$ across depth and measure linear and small-MLP probe accuracy for speaker (SLURP and LibriSpeech speakers) and emotion (CREMA-D). This tests whether DAISY's shallow-identity finding holds for an intent-tuned model and picks the deepest usable layer. We then estimate $P$ and verify at the teacher that intent accuracy is retained and probes fall to chance.

Students come from FRILL's MobileNetV3 family with global pooling replaced by lightweight temporal convolution. The replacement is needed because Representation Based Meta-Learning found that mean pooling loses order on 3-second commands. Students are 8-bit quantized and sized to fit under SILENCE's 394.9 KB. We train with the loss above, and with a teacher-assistant stage if the size gap proves too large. Latency and memory are measured on an STM32H7-class board.

Privacy is evaluated only against informed attackers. These attackers see the student's output features for enrollment as well as test data, which mirrors the anonymized-enrollment setting under which Introducing the VoicePrivacy Initiative found identity leakage (EER falling from 47–58% to 26–37%). There are three attackers. The first is a speaker-verification model trained on features, reported as equal error rate. The second is a nonlinear emotion classifier. The third is an inversion decoder trained to reconstruct spectrograms, which are then vocoded, transcribed, and scored for word error rate and for speaker-verification EER against the original speaker.

Baselines

The naive reference is sending raw audio to a cloud intent model. SILENCE reports its 81.2% SLURP intent accuracy as about 7% below this raw-audio setting. We re-measure the raw-audio setting with our cloud model to get its exact accuracy and its (maximal) leakage.

The strongest existing method is SILENCE: 81.2% intent accuracy on SLURP and over 80% malicious-ASR word error rate, reported in SILENCE. We run it on the same splits and the same microcontroller, pair it with a strong cloud intent model, and give it the same hyperparameter-tuning budget as our method. We then apply our speaker, emotion, and inversion attacks to its masked output, which the original paper did not evaluate. Beating SILENCE means matching or exceeding 81.2% intent accuracy within its memory and latency envelope while pushing informed-attacker speaker EER toward 50% and emotion accuracy toward chance, where SILENCE is expected to leak.

The on-device discrete-unit extractor from On-device Streaming Discrete Speech Units is run with an intent head on the same data. It is the main competitor on the privacy axis. Its compute is far above microcontroller scale (0.96 TFLOPs per minute), so we compare it at its own cost and also at a unit-count setting pruned toward our budget. No reported speaker-verification EER was found on Lacuna for discrete speech units.

As a cascade reference, SLURP reports 81.42% action accuracy on noisy ASR transcripts versus 86.99% on gold transcripts. We treat this as the privacy ceiling that text gives, and we note its compute is far above microcontroller scale.

Three ablations isolate the key ingredient. Each keeps student architecture, data, and tuning budget identical. The first is a student trained on intent labels only, with no teacher. The second is a student distilled from the unscrubbed target $h_L$, which is FRILL-style distillation. The third is the task-only student plus a gradient-reversal speaker adversary. The proposal beats these ablations only if the scrubbed-target student sits strictly above all of them on the curve of intent accuracy against attacker EER, at matched memory.

Why not just…?

Why not just train the tiny model on intent labels and skip the teacher? This is the most serious threat. A small network trained only to classify intent may discard identity on its own. If so, distillation adds accuracy but not privacy. The label-only ablation decides this. If its attacker EER matches that of the scrubbed-target student, we will report that task supervision is enough for privacy and reframe the contribution as distillation for utility at microcontroller scale. We expect label-only students to leak more, because nothing penalizes speaker cues that happen to help with the training speakers. But we do not assume the result.

Why not just add an adversarial speaker loss? Adversarial removal is known to leave features that a fresh attacker can still exploit, and our informed-attacker protocol is designed to expose exactly that. Scrubbing the target is a fixed, non-adversarial objective that is cheaper and more stable to train. The adversarial ablation measures whether it is also more private. Combining both is a fallback if neither dominates.

Why not just use SILENCE's masking? Masking keeps raw frames. The frames that carry intent also carry voice, and SILENCE reports only transcription failure. If our attacks find that SILENCE's output already resists speaker verification and emotion inference, most of our privacy advantage disappears, and the remaining question is purely one of cost.

Why not just use discrete speech units? Their identity removal is asserted, not measured, and their extractor is orders of magnitude above microcontroller compute. If pruned discrete units reach our leakage level within our memory budget, they win on simplicity, and we will say so.

Decisive Checks

We evaluate on SLURP and Fluent Speech Commands for intent, LibriSpeech and SLURP speakers for speaker verification, and CREMA-D for emotion. Four results settle the claim. First, at the teacher, removing the probe subspace from the deep layer must keep intent accuracy within a few points while bringing linear probes to chance. If it does not, the target idea fails before any student is trained. Second, the scrubbed student's nonlinear-attacker leakage must be no higher than the teacher target's leakage measured with the same attacker. Leakage above the target's would mean the student re-learned identity from the audio. Third, at matched memory, the scrubbed student must beat the label-only and adversarial students on the accuracy-versus-EER curve. Fourth, on hardware, it must fit SILENCE's envelope while matching its 81.2% intent accuracy and exceeding its measured resistance to speaker and emotion attacks.