Speech emotion recognition means inferring a speaker's emotional state, such as angry, sad or neutral, from the sound of their voice. The best current systems fine-tune large pretrained speech encoders like wav2vec 2.0 or HuBERT on a few hours of emotion-labeled audio. Those continuous speech representations are known to be fragile, and physical attacks on speech pipelines are real: inaudible ultrasonic perturbations reach 100% success against command recognition and survive compression and voice-activity filtering (Inaudible Adversarial Perturbation). The standard fix is adversarial training, which retrains the model on worst-case perturbed inputs. That fix hits two walls at once in this setting. It multiplies the cost of training a large encoder by the number of attack steps. It also needs many labeled examples, and with only an hour of emotional speech it tends to memorize the training perturbations and lose clean accuracy.
This note proposes to separate robustness from emotion labels entirely. We freeze the speech encoder and append a handful of learnable "robustness tokens", extra input vectors the transformer attends to. We train only these tokens, and only on unlabeled speech: the objective asks that attacked audio produce the same internal features as clean audio, and that clean features stay unchanged. The scarce emotion labels are then used only to train a light classifier on clean features. The expensive attack loop never touches the labeled set, runs once, and can be reused across downstream tasks. The risk specific to emotion is that tokens trained to ignore small acoustic changes may also learn to ignore prosody, the pitch, energy and timing cues that carry emotion. Subtle low-arousal emotions such as sadness would suffer most, since they already sit close to neutral speech. We add an explicit term that preserves prosodic contrasts and test whether it closes that gap.
Core Claim
The claim is that a robustness module learned without any emotion labels, on a frozen speech encoder, gives robust accuracy comparable to or better than full adversarial fine-tuning on the labeled set. We expect this to hold especially at 30–60 minutes of labeled audio, and at a small fraction of the compute. We also expect it to keep clean recall on low-arousal classes, provided the module is explicitly constrained to preserve prosodic differences.
The method comes from Robustness Tokens. That paper appends 10 learnable tokens to a frozen Vision Transformer and optimizes them with a clean-invariance loss plus a loss that pulls attacked features back to clean ones. On DINOv2-G it recovers white-box ImageNet accuracy from 0.3% to 63.1%, with clean accuracy at 85.2% versus 85.6%, after 200 steps. Its losses are feature-level and never use class labels. That property is what makes it attractive under emotional data scarcity, but the paper never exploits it: it trains and evaluates on vision benchmarks only. We keep the mechanism and change three things: the modality (waveform attacks through a speech encoder), the training data (unlabeled speech that differs from the labeled task), and the objective (an added prosody-preservation term). We do not claim the token idea itself as new.
Let $g(x)$ be the frozen encoder's layer-wise features for waveform $x$, and $g_T(x)$ the same features with tokens $T$ appended. Let $x'$ be a natural, audible prosodic variant of $x$ produced by pitch shift, tempo change or energy scaling. The tokens minimize
$$\mathcal{L}(T)=\mathbb{E}_x\Big[\|g_T(x)-g(x)\|^2+\max_{\|\delta\|\le\epsilon}\|g_T(x+\delta)-g(x)\|^2+\lambda\,\|(g_T(x')-g_T(x))-(g(x')-g(x))\|^2\Big],$$
Here $\delta$ is a waveform perturbation bounded by $\epsilon$, found with projected gradient descent, which is repeated gradient steps projected back into the allowed perturbation size. The weight $\lambda$ trades off prosody preservation. The first two terms follow the robustness-token recipe. The third requires that large, meaningful prosodic changes move the representation as much as they did before the tokens were added. Without it, the cheapest way for the tokens to cancel small perturbations may be to flatten sensitivity to acoustic variation in general.
Why This Intersection, and What Prior Work Leaves Open
Three strands motivate the proposal. The first is fragility. Deep Architecture Enhancing Robustness is the only work in the file that attacks an emotion recognizer. It uses mixup, which trains on blended pairs of examples and their labels, on a small CNN-LSTM. It reports 41.4% unweighted average recall under one-step FGSM, versus 36.1% without mixup, and 64.1% clean. It never uses iterative attacks, foundation encoders or true adversarial training. The second is scale. Full adversarial retraining of large models is described as impractical (How Robust is Google's Bard). Cheap alternatives, including robustness tokens, the "Adversarial Possibility" text prompt that lifts LLaVA-7B from 12.54% to 91.46% under Auto-PGD on COCO (Improving Adversarial Robustness in VLMs), and attention rectification in text transformers (Dynamic Attention), have not been tested on speech. The third is data scarcity. Strong emotion systems such as P-TAPT (74.3% unweighted accuracy on IEMOCAP, still strong with 30–60 minutes of labels), Emotion Prompting (79.52% weighted accuracy, 5-fold) and A-SMiLE, a frozen HuBERT adapter, were never attacked.
Two older results constrain the design. Using neutral speech models models emotion as a deviation from a neutral-speech model trained on abundant neutral data. In that framing, low-arousal negative speech overlaps with neutral: only 42% of negative call-center utterances are recognized. This is exactly the region where a smoothing defense would do damage, and it is why we report per-class recall rather than averages alone. On the defense side, Adversarial Neuron Pruning and Training set cleansing show that defenses erode rare or subtle features first, so we do not assume the vision result carries over to affect. Speaker-invariant Affective Representation Learning uses "adversarial training" in a different sense: it removes speaker identity from the features (59.91% on IEMOCAP). It says nothing about robustness to crafted input perturbations. We treat it as orthogonal, not as a competitor.
Research Program
We would build the token module for frozen wav2vec 2.0 Base and Large, HuBERT Large, and a P-TAPT-adapted wav2vec 2.0. The tokens are inserted into the transformer's input sequence after the convolutional front end, and attacks are applied on the raw waveform so the front end is included in every gradient. The perturbation budget $\epsilon$ is defined by signal-to-noise ratio, sweeping 30–50 dB.
The token training data is the variable we care about. We compare three sources. The first is neutral read speech from LibriSpeech, the "neutral anchor" in the sense of Busso et al. The second is unlabeled mixed conversational speech from CommonVoice and GigaSpeech. The third is unlabeled in-domain audio, meaning IEMOCAP or MSP-IMPROV training audio with labels discarded.
For the analysis, we will test whether the "massive activations" mechanism reported in vision holds in speech encoders. In vision, attacks suppress a few outlier feature values that act as internal biases, and the tokens restore them. We will also test whether attacks concentrate attention on a few frames, the "attention hijacking" pattern seen in text. Each finding would say whether the defense is a transfer of a known mechanism or something speech-specific.
Baselines
The naive reference is the undefended system: the same frozen encoder and emotion classifier with no tokens, evaluated clean and under attack. This bounds the gain and the clean cost.
The strongest competitor is full-parameter adversarial fine-tuning of the encoder plus classifier on the labeled emotion set. Each training example is replaced by a projected-gradient-descent adversarial version, using a loss that balances clean and adversarial accuracy. This directly optimizes the target metric with labels, so if labels were not the bottleneck it should win. We run it with the same encoder initialization and the same attack budget. We give it at least the tuning budget the tokens receive, a sweep over learning rate, attack steps and the clean/adversarial weight, selected on held-out speakers. We report GPU-hours for both methods. On clean accuracy the reference points are 74.3% unweighted accuracy for P-TAPT and 79.52% weighted accuracy for Emotion Prompting on IEMOCAP. For robustness to iterative attacks on speech-foundation emotion models: no reported result found on Lacuna for PGD or Auto-PGD accuracy on IEMOCAP. The only adversarial number is Latif et al.'s 41.4% recall under FGSM, which we reproduce as a weak-attack anchor. Beating this baseline means robust unweighted accuracy at least equal at full data and higher at 30–60 minutes, a clean drop no larger than adversarial fine-tuning's, and at least 5x less compute.
A cheaper competitor is adversarial fine-tuning with a parameter-efficient adapter, such as a small bottleneck or LoRA-style update on a frozen encoder, trained with labeled PGD. It matches the tokens' parameter count and removes the "fewer parameters" confound, so any remaining gap must come from label-free training or the prosody term. We give it the same parameter count, data and tuning budget.
We also run mixup following Latif et al. on the same encoder, and the same prosody-preserving tokens trained on the labeled set with its labels ignored. The latter separates "unlabeled data at scale" from "any feature-level objective".
Two ablations isolate the proposal's ingredients. Removing the prosody term ($\lambda=0$) gives the plain robustness-token recipe. We count the idea as supported only if the full objective beats it on clean recall for sad versus neutral without losing robust accuracy. The data-source ablation (neutral, mixed, in-domain unlabeled) tells us whether neutral speech is sufficient or whether the tokens must see expressive speech.
Why not just…?
Why not just adversarially fine-tune the whole model on the labeled data? This is the real threat, and the proposal is built to lose honestly to it if it deserves to. Our reason for expecting it to struggle is that attack-based training on a few hundred utterances tends to overfit the training perturbations. We have not measured this for speech emotion; the file reports no such experiment. The labeled-data sweep at 30 min, 1 h, 3 h and the full set settles it. If full adversarial fine-tuning matches the tokens at 30–60 minutes at comparable clean accuracy, the label-free argument fails and the remaining contribution is compute savings only.
Why not just use mixup or other augmentation? The only emotion-recognition evidence, a gain from 36.1% to 41.4% recall (Deep Architecture Enhancing Robustness), comes from one-step FGSM. That attack is known to overstate robustness when gradients are obscured. Aggressive augmentation also destabilizes training and costs clean accuracy (Color Backdoor). We run mixup on the same encoder under iterative and adaptive attacks. If it holds up there, it is the simpler answer and we will report that.
Why not just filter or compress the audio before the model? Quantization, voice-activity filtering and Opus compression all failed against realistic speech attacks, which kept 100% success (Inaudible Adversarial Perturbation). Input filters also reject or distort benign unusual inputs (Baseline Defenses), and in emotion recognition unusual prosody is often the signal. We include low-pass filtering and Opus re-encoding as baselines under adaptive attacks that differentiate through them or approximate them.
Why not just prepend a fixed "this may be adversarial" audio prompt? That works in vision-language models because a language model reasons over the prompt. A speech encoder feeding a classifier has no such reasoner. Emotion Prompting's audio prompts were trained jointly and were never attacked. We test a synthesized-speech defensive prompt at zero training cost as a baseline, and expect it to do little.
Proposed Experiments
We use IEMOCAP 4-class with leave-one-session-out evaluation at 30 min, 1 h, 3 h and full labeled data. Cross-corpus transfer is IEMOCAP to MSP-IMPROV, where Latif et al. report 46.81% clean recall as a reference. We report clean unweighted accuracy, per-class recall with emphasis on sad and neutral, and robust accuracy under 100-step PGD and Auto-PGD across the signal-to-noise budget sweep. We also report GPU-hours and total forward and backward passes. Every defense is attacked with full knowledge of the tokens, differentiating through them, as well as in the token-secret setting. Robustness tokens have only been evaluated with the tokens kept secret, and we will not report a number that depends on secrecy without labeling it.
Four outcomes are decisive. First, the label-free tokens must reach at least the robust accuracy of full adversarial fine-tuning at 1 hour of labels, at much lower compute. Second, the prosody term must recover sad-versus-neutral clean recall relative to $\lambda=0$. Third, neutral-only LibriSpeech tokens must perform within a few points of in-domain unlabeled tokens. If they do, one module trained on neutral speech could protect many affective tasks. If they do not, the defense requires expressive speech. Fourth, gains must survive the tokens-known attack. If the tokens-known attack collapses them, as the secrecy concern in the vision work suggests may happen, the result is a negative finding about this class of defenses for speech. That finding would still be useful to report.
References
- Inaudible Adversarial Perturbation: Manipulating the Recognition of User Speech in Real Time
- Robustness Tokens: Towards Adversarial Robustness of Transformers
- Deep Architecture Enhancing Robustness to Noise, Adversarial Attacks, and Cross-Corpus Setting for Speech Emotion Recognition
- How Robust is Google's Bard to Adversarial Image Attacks?
- Improving Adversarial Robustness in Vision-Language Models with Architecture and Prompt Design
- Improving the Robustness of Transformer-based Large Language Models with Dynamic Attention
- Exploring Wav2vec 2.0 Fine Tuning for Improved Speech Emotion Recognition
- Emotion Prompting for Speech Emotion Recognition
- A-SMiLE: Affective Sparse Mixture-of-Experts Adapter with Multi-Task Learning for Spoken Dialogue Models
- Using neutral speech models for emotional speech analysis
- Adversarial Neuron Pruning Purifies Backdoored Deep Models
- Training Set Cleansing of Backdoor Poisoning by Self-Supervised Representation Learning
- Speaker-Invariant Affective Representation Learning via Adversarial Training
- Color Backdoor: A Robust Poisoning Attack in Color Space
- Baseline Defenses for Adversarial Attacks Against Aligned Language Models