Guard Tokens for Speech Foundation Models

A speech model can be attacked in two very different ways, and current defenses handle at most one of them. At test time, an adversary adds noise so quiet that no listener hears it, yet the model transcribes silence instead of speech — Muting Whisper shows that a single 0.64-second prefix, optimized once, silences over 97% of unseen LibriSpeech audio for Whisper. At training time, an adversary poisons the data or the model so that a hidden trigger flips the model's behavior while everything else looks normal: CBA plants sample-specific compression triggers that survive every standard filter and pruning defense, and Backdoor Attacks Against Speech Language Models shows that when a speech encoder is poisoned, the backdoor propagates into every downstream module. The defenses that exist either cost a fortune (full retraining or adversarial training of a multi-hundred-million-parameter model) or quietly damage normal performance — purified audio gets its benign phonemes corrupted, and misflagged clean speech loses up to 10% of its accuracy.

The proposal: instead of retraining or filtering anything, prepend a handful of small learnable vectors — "guard tokens" — to the input of a frozen speech encoder, and train only those tokens. This transfers the parameter-efficient defense of Robustness Tokens: Towards Adversarial Robustness of Transformers, which restores robust accuracy in vision transformers (from 0.3% to 63.1% under white-box attack on ImageNet with DINOv2-G) at a cost of about 10 tokens and 200 training steps, into the setting where it has never been tried: audio, sequence-to-sequence speech models, and — the genuinely new part — training-time backdoors, which the original paper explicitly never examined.

Core Claim

A set of roughly ten learnable prefix tokens, inserted into a frozen speech encoder (Whisper, Conformer, WavLM), trained with a combined objective on clean, adversarially perturbed, and trigger-carrying audio, can simultaneously (a) block test-time imperceptible perturbations by competing for the attention that ungrounded acoustic prefixes capture, and (b) suppress backdoor pathways by forcing trigger-carrying inputs to map to the same representations as their clean counterparts — all while keeping clean accuracy within a fraction of a percent of the undefended model, training only ~0.1% of parameters, and adding negligible inference latency.

The key mechanism is representational: both attacks we target work by hijacking the encoder's representation space. The Muting Whisper prefix works because special tokens like <|endoftext|> have no acoustic grounding, so cross-attention heads get captured by an artificial input that maps straight into that ungrounded latent slot; CBA-style backdoors work because poisoned training isolates conditional logic in spare encoder capacity that clean data never touches, exactly as Fine-Pruning documents. Guard tokens attack the common substrate: they add a small, persistent, input-independent component to the encoder state that (i) occupies attention mass an attacker's prefix would otherwise monopolize and (ii) is optimized to make the encoder's output invariant to trigger presence.

Concretely, with $z_G(x)$ the encoder's pooled output for utterance $x$ when the guard tokens $G$ are prepended:

$$\mathcal{L}(G) = \|z_G(x) - z(x)\|_2^2 + \lambda_a \|z_G(x + \delta) - z_G(x)\|_2^2 + \lambda_b \|z_G(t(x)) - z_G(x)\|_2^2$$

where $z(x)$ is the frozen encoder's output without tokens, $\delta$ is an adversarial perturbation, and $t(x)$ is the same utterance carrying an attack trigger. The first term preserves clean behavior, the second corrects adversarial drift (the same dual-objective structure as Robustness Tokens), and the third — the new ingredient — suppresses the backdoor by making the triggered and clean representations coincide, so the downstream decoder never sees the attacker's conditional input.

Research Program

Three components, in escalating order of difficulty. First, port and instrument the guard-token mechanism for audio: decide where tokens enter (encoder input embeddings versus decoder cross-attention), how representations are pooled for the alignment losses in a sequence-to-sequence model (Robustness Tokens was evaluated only on classification and linear probes, so this is uncharted), and how many tokens suffice. Second, train and evaluate against the two attack families separately, then jointly, to measure whether one objective interferes with the other — the robustness-accuracy trade-off is the central scientific question, and we measure it as the clean Word Error Rate penalty paid per point of attack suppression. Third, run adaptive attacks, because the guard tokens are a "private key": Robustness Tokens concedes that token extraction defeats the defense, so we must measure degradation when the attacker knows the defense exists, has query access to the token-equipped model, and re-optimizes triggers (including a CBA-style attack run against the defended model directly).

Baselines

The naive reference point is the undefended frozen model: on Google Speech Commands with 5% poisoning, CBA-Bit on ResNet-34 reports an attack success rate of 99.73% with 96.69% clean accuracy (CBA); on LibriSpeech, the Muting Whisper prefix silences over 97% of unseen samples, collapsing average output length from roughly 18 words to between 0.04 and 0.56 (Muting Whisper); in the SpeechLLM setting, a poisoned WavLM encoder keeps a 2.13% clean WER while achieving 99.20% attack effectiveness, with encoder-only AER reaching 90.76–99.41% across tasks (Backdoor Attacks Against Speech Language Models). These numbers define what "broken" looks like before any defense.

The strongest existing competitor is AudioPure (Defending against Adversarial Audio via Diffusion Model), an inference-time diffusion purifier that needs no retraining and reports roughly 84% robust accuracy on SC09 under white-box attack, beating adversarial training by up to 20%. It will be run on the same frozen backbones, the same attack implementations, and with the same tuning budget (its reverse-step count $n^*$ swept on a validation split exactly as we sweep our token count and loss weights). Beating AudioPure means matching or exceeding its attack suppression while cutting its costs: its purification takes about one second of RTX 3090 time per second of audio and suffers clean degradation when $n^*$ overshoots, so we would count a win as comparable suppression with under 0.1× inference overhead and a clean-accuracy penalty within one point of the undefended model. The second competitor is SpeechGuard (SpeechGuard: Online Defense against Backdoor Attacks on Speech Recognition Models), whose S-STRIP detection plus time-frequency masking drops backdoor success by over 90% but degrades clean accuracy by up to 10% and to roughly 60% under random-noise triggers; matching its ≥90% attack reduction with under 1% clean loss would be a decisive result. Two weaker baselines bracket the design space: Fine-Pruning pruning of least-active encoder units (CBA reports no pruning point that neutralizes its backdoor without destroying accuracy), and clean-data fine-tuning with LoRA adapters, which the SpeechLLM paper shows fails against a poisoned encoder. Finally, the key ablation isolates our new ingredient: guard tokens trained with only the invariance and adversarial terms (no backdoor term). If that variant suppresses backdoors as well as the full objective, the backdoor-specific loss — and the paper's central claim — is unsupported.

Why not just…?

Why not just low-pass or Wiener filter the audio? This is the zero-cost answer a reviewer will reach for first, and the evidence against it is direct: AudioPure reports that traditional transformation defenses collapse to near 0% robust accuracy under adaptive white-box attacks, and CBA's compression triggers are explicitly built to mimic speech structure and demonstrably resist low-pass, Gaussian, median, and Wiener filtering (CBA; AudioPure). We will still run these filters as sanity baselines so the comparison is on the record.

Why not just prune the backdoored encoder? Neuron pruning is the classic post-hoc answer, but Adversarial Neuron Pruning Purifies Backdoored Deep Models shows that pruning sensitive neurons also destroys legitimate feature channels, and CBA reports no pruning point that removes its trigger without breaking clean classification. Our approach touches no encoder weights at all, which is precisely why the clean-accuracy risk is smaller.

Why not just fine-tune with LoRA on clean data? This is the most serious objection because it is parameter-efficient too. The SpeechLLM paper already tested it: partial fine-tuning fails to remove the backdoor from the poisoned encoder, leaving near-100% attack effectiveness downstream (Backdoor Attacks Against Speech Language Models). The likely reason is that adapters change how representations are consumed downstream without erasing the conditional pathway inside the encoder; our loss instead explicitly forces triggered and clean representations to coincide inside the encoder. This difference is testable and is exactly what the LoRA baseline in our experiments will measure.

Why not just purify with a diffusion model instead of training anything? Honest answer: for input-trigger attacks this is a real threat, and AudioPure is the strongest baseline precisely for that reason. It falls short on three measurable axes: roughly one second of GPU compute per second of audio (unsuitable for streaming), a documented clean-accuracy "sweet spot" dilemma where extra reverse steps hallucinate phonemes, and no evaluation against backdoors at all — and it cannot, even in principle, address a poisoned encoder whose malicious logic lives in the weights rather than in the input, since removing the trigger from one input leaves the pathway armed for the next. Our experiments will run AudioPure on every input-trigger attack so the head-to-head is empirical rather than asserted; if it matches our suppression at acceptable cost, the contribution narrows honestly to the backdoor and latency settings.

Why not just noise-augmented training? Adding random noise during training is cheaper than min-max adversarial training, but Comparative study on noise-augmented training and its effect on adversarial robustness in ASR systems finds it does not reliably confer adversarial robustness across ASR architectures, and aggressive augmentation itself degrades clean accuracy (Color Backdoor documents this failure mode in vision). More fundamentally, it requires training, which is the cost we are trying to avoid.

Proposed Experiments

Experiment one targets keyword classification on Google Speech Commands with ResNet-34 and a small speech transformer, replicating the CBA protocol at 5% poisoning with both Bit and Wavelet compression triggers. Success means attack success rate below 10% (from 99.73%) with clean accuracy within one point of 96.69%, at a training cost comparable to the ~200 steps and <0.15M-parameter scale of SpeechPrompt v2 prompt tuning.

Experiment two moves to sequence-to-sequence ASR: Whisper and Conformer on LibriSpeech, attacked with the Muting Whisper universal prefix under the same $l_\infty$, $\epsilon = 0.02$ budget, plus gradient-based per-sample attacks. Decisive metrics are average output sequence length recovering from the 0.04–0.56 range toward the ~18-word baseline, clean WER within about 0.3 absolute points of the undefended model (Conformer Large's reference point is 2.1%/4.3% WER without a language model, Conformer), and transfer testing on TED-LIUM3 and MGB where the original attack retains over 90% success.

Experiment three reproduces the SpeechLLM "Patient Zero" setting: a poisoned WavLM or Whisper encoder plugged into a clean LLM, evaluated on LibriSpeech ASR, VoxCeleb2-AE speaker profiling, and CREMA-D emotion recognition, where the undefended AER is 90.76–99.41%. Guard tokens are trained on the poisoned encoder (simulating a defender who suspects contamination but cannot afford full retraining) and we measure whether AER falls below 10% while clean WER stays near the 2.13% poisoned-but-benign reference.

Experiment four is the decisive check on honesty: adaptive attacks throughout, including an attacker with white-box access to the guard tokens, a CBA-style trigger re-optimized against the defended model, and a trigger distilled to survive representation alignment. The proposal survives only if suppression degrades gracefully — say, attack success stays below 30% under full adaptive knowledge — rather than collapsing to the undefended 99%+. The private-key weakness of Robustness Tokens is a known limitation; quantifying it in the audio-backdoor setting, and testing mitigation by per-deployment token re-randomization, is itself a contribution the literature does not yet contain.