Speech emotion recognizers are increasingly built in two pieces. A large pretrained speech model turns audio into a vector, and a separate classifier reads emotion off that vector. When that classifier is a random forest or gradient-boosted trees, it has no gradient, so standard attacks have nothing to follow. In vision, swapping a network's final layer for a random forest was reported to reduce one white-box attack's success to zero at almost no cost in clean accuracy. That is a striking claim. It is also exactly the kind of result that often turns out to be gradient masking: the attack fails because it cannot see the slope, not because the model is hard to fool. In speech, the feature extractor in front of the trees is still fully differentiable. We propose to test which explanation holds, using attacks built to defeat masking.
The second question asks which utterances such attacks break and which ones a defense sacrifices. Emotion data comes in two flavours. Acted speech is exaggerated and easy to label. Spontaneous speech is subtle, and raters often disagree about it. The plausible hypothesis is that vulnerability, and the clean-accuracy cost of any defense, concentrate on ambiguous utterances. The interesting twist is that in IEMOCAP, the standard acted-dialogue corpus, improvised turns are more agreed-upon than scripted ones. That lets us separate "spontaneous" from "ambiguous" as explanations. In two sentences: we check whether tree-based emotion heads give real adversarial robustness or only hide gradients, and we measure whether attack success and defense cost track how ambiguous an utterance is rather than whether it was acted.
Core Claim
Our central prediction concerns frozen speech-model embeddings (wav2vec 2.0, HuBERT, and the emotion-tuned wav2vec2-large-robust). On these embeddings, a random-forest or gradient-boosted head will look robust against direct gradient attacks and naive black-box attacks. Its robustness will largely disappear under adaptive attacks of two kinds. The first differentiates through a smooth surrogate of the trees. The second moves the embedding directly toward another class's region. The reason is that the perturbation is applied to audio, and the audio-to-embedding map is where the fragility lives.
Our second prediction is that, across corpora, the minimal perturbation needed to flip a prediction shrinks with human rater disagreement. Once agreement is controlled for, the acted-versus-improvised label explains little additional variance. Likewise, the clean accuracy lost to adversarial training of the backbone falls disproportionately on low-agreement utterances.
Either prediction can fail informatively. If tree heads survive adaptive attacks, that is the first evidence of genuine non-gradient robustness in audio, and it would justify the hybrid design. If elicitation type predicts fragility beyond agreement, then acted training data carries attack-relevant shortcuts that ambiguity alone does not explain.
What prior work shows and does not
The motivating result is Defending Against Adversarial Attacks Using Random Forest. It replaces the upper layers of AlexNet/VGG with a random forest on MNIST and CIFAR-10. It reports 0% Carlini–Wagner success, zeroth-order (ZOO) success falling from about 90% to under 6%, and clean accuracy within 0.5–2.9% of the original network. It never tests speech. It also does not report attacks designed for a non-differentiable head, and that file itself lists this as open: whether surrogate or zero-order attackers can approximate the tree boundary. We reuse its architecture and supply the missing adaptive evaluation.
Two speech papers already pair learned embeddings with a statistical classifier, but neither attacks the result. EmoDB 2.0 trains an RBF-kernel SVM on wav2vec2-large-robust embeddings and eGeMAPS features. It shows that training on low-agreement samples helps, and that agreement correlates with perceived naturalness (r = .816). Using denoising autoencoder for emotion recognition feeds autoencoder features to an SVM on IEMOCAP (61.5% accuracy versus 58.6% for IS2010 features) and documents confusion between sadness and neutral speech. These papers supply our data strata and hybrid baselines. They also suggest where small margins should appear.
The only adversarial speech emotion work in the file is Deep Architecture Enhancing Robustness to Noise, Adversarial Attacks, and Cross-Corpus Setting. It uses a softmax DenseNet-LSTM-Highway model with mixup. It reaches 64.1% unweighted average recall (UAR, mean per-class recall) clean, 41.4% under single-step FGSM (versus 36.1% for its baseline), and 46.81% when tested cross-corpus on MSP-IMPROV. It uses only single-step attacks and does not split results by elicitation type. That constrains us: mixup alone leaves a large gap, so it is a baseline, not a solution.
Parameter Efficient Finetuning for SER and Domain Adaptation shows that small adapters on wav2vec 2.0 reach 67.00% IEMOCAP accuracy, against 66.30% for full fine-tuning. Scripted-to-improvised adaptation of HuBERT gives roughly 6% more. This supplies the strongest differentiable model, and a cheap way to adversarially train the backbone.
The elicitation side rests on three findings. Scripted dialogs versus improvisation reports 83.1% agreement on improvised turns versus 66.9% on scripted ones. Real vs. acted emotional speech and the articulatory work of Narayanan et al. show that acted portrayals are more intense and physically exaggerated. Together these let us pull apart two competing explanations for fragility: exaggeration-as-shortcut and ambiguity-as-small-margin.
Correlation Analysis of Adversarial Attack in Time Series Classification supplies a method. It shows that 1-D classifiers over-rely on high-frequency local structure, with 0.908 attack success for an FFT-concentrated attack on ResNet18, and that filtering defenses cost clean accuracy. We borrow its band-limited attack to ask whether speech-model embeddings share the same high-frequency weakness.
Research Program
We build three families of classifiers on identical frozen embeddings, mean-pooled from each layer and from a learned layer-weighted sum. The first family is differentiable heads: a linear probe and a small MLP. The second is hybrid heads: random forest, gradient-boosted trees, and RBF SVM. The third family, built only for analysis, is smooth surrogates of each tree model. These are soft decision trees or MLPs distilled to match the forest's predicted probabilities on training and augmented embeddings. Separately, we train PEFT-adapted end-to-end models with and without adversarial training (PGD on the waveform at a fixed signal-to-noise budget). This tells us whether hardening the backbone, rather than swapping the head, is what buys robustness.
Attacks act on the raw waveform under an L∞ budget, expressed as signal-to-perturbation ratio so that budgets are interpretable across speakers. They escalate in strength.
- FGSM and multi-step PGD straight through the model. Against hybrid heads these fail by construction.
- Hard-label and score-based query attacks in the spirit of ZOO.
- Transfer from a softmax head trained on the same frozen backbone.
- Two adaptive attacks. One runs PGD through backbone plus distilled surrogate. The other is a feature-space attack that minimizes the distance between the perturbed embedding and the training embeddings of a target class, with no reference to the head at all.
- The band-limited FFT variant of the strongest attack, following Li et al.
The headline measurement is robust UAR under the strongest of these. We also record, for each utterance, the minimal budget at which any attack succeeds. This number serves as an empirical margin.
We stratify by two independent axes. One is elicitation: IEMOCAP scripted versus improvised, plus acted EmoDB 2.0 versus naturalistic MSP-IMPROV transfer. The other is rater agreement, using IEMOCAP's evaluator consensus and EmoDB 2.0's agreement and naturalness scores. We then regress log minimal perturbation on agreement, elicitation, emotion class, and speaker. For the trade-off question, we decompose the clean UAR lost to adversarial training by stratum, and ask whether low-agreement bins absorb more than their share.
Baselines
The do-nothing reference is the undefended linear probe and the undefended PEFT model on the same embeddings and folds. These anchor clean accuracy and attack success. The strongest reported clean model in the file is PEFT-adapted wav2vec 2.0 at 67.00% IEMOCAP accuracy in Parameter Efficient Finetuning for SER. We reproduce it under leave-one-session-out cross-validation and report UAR.
The strongest reported adversarial speech emotion baseline is mixup training from Latif et al., at 41.4% UAR under FGSM against 64.1% clean. We re-implement mixup on our PEFT backbone rather than on their DenseNet, so that architecture differences do not confound the comparison. We evaluate it under the same multi-step and adaptive attacks as every other model.
The strongest hybrid baseline is the Ding et al. recipe ported to audio. In vision it reports 0% C&W success and clean accuracy within 0.5–2.9% of the network. No reported result was found on Lacuna for any tree-headed speech emotion model under attack.
The key ablation is a head swap with everything else held fixed: the same embeddings, folds, and attack budgets, with a linear probe against a random forest. This isolates whether the head contributes anything once attacks are adaptive.
Every model gets the same hyperparameter search budget, tuned on clean validation UAR only. Every attack gets the same query and iteration budget, so that no defense wins by starving its attacker.
A hybrid head "beats" adversarially trained PEFT only under one condition: higher robust UAR under the strongest adaptive attack at matched budget, with clean UAR no more than one standard deviation (across folds) below. Robustness against naive attacks alone does not count.
Why not just…?
Why not just use a linear probe and skip the trees? This is the real threat, and our main hypothesis is that it is correct: if fragility sits in the backbone, the head is irrelevant. The head-swap ablation under adaptive attacks settles it directly. If linear and tree heads have the same robust UAR, the hybrid design gives no security benefit in speech, and we will report that.
Why not just adversarially train the adapters? This may well be the best defense. Direction 3 of the file predicts it costs clean accuracy, and no number exists for speech emotion. We run it as a full competitor, and its clean-accuracy loss is the quantity we decompose by agreement stratum. So it is part of the experiment, not something we argue against.
Why not just use mixup or noise augmentation, as Latif et al. did? Their own numbers show a 64.1% to 41.4% UAR drop under single-step FGSM. That figure is likely optimistic, since multi-step attacks are stronger. Li et al. also report that filtering and noise augmentation cost clean accuracy on time-series models. We include mixup under the full attack suite so the comparison is direct.
Why not just use hand-crafted eGeMAPS features with an SVM, which has no deep gradients? The deep-embedding SVM beat IS2010 features (61.5% versus 58.6%) in Xia & Liu. EmoDB 2.0 shows standard acoustic descriptors miss information that glottograms add. More importantly, eGeMAPS features are themselves computable functions of the waveform. We attack them through a differentiable reimplementation of pitch and energy features, or through transfer, so that "no gradients" is tested rather than assumed.
Decisive Experiments
The study rests on three checks.
The first is on IEMOCAP (leave-one-session-out, four classes) at budgets around the quietest audible perturbation. If the surrogate or feature-space attack drives tree-head attack success to within about ten points of the linear head's, the vision result does not carry over and is best read as masking. If a large gap survives every adaptive attack, hybrid heads earn a real place in robust speech emotion recognition.
The second is the margin regression on IEMOCAP and EmoDB 2.0. We expect agreement to carry most of the explained variance in minimal perturbation, and the scripted/improvised label to add little. Improvised IEMOCAP turns, being higher-agreement, should then be more robust than scripted ones, the opposite of the naive "spontaneous is fragile" story. The reverse outcome would point to acted-speech shortcuts.
The third is cross-corpus. We train on IEMOCAP, test clean and attacked on MSP-IMPROV, and compare against the 46.81% clean transfer UAR of Latif et al. This shows whether the agreement–margin relationship and the stratified clean cost hold under the acted-to-naturalistic shift that deployed systems actually face.
References
- Defending Against Adversarial Attacks Using Random Forest
- EmoDB 2.0: A Database of Emotional Speech in a World that is not Black or White but Grey
- Using denoising autoencoder for emotion recognition
- Deep Architecture Enhancing Robustness to Noise, Adversarial Attacks, and Cross-Corpus Setting for Speech Emotion Recognition
- Parameter Efficient Finetuning for Speech Emotion Recognition and Domain Adaptation
- Scripted dialogs versus improvisation: lessons learned about emotional elicitation techniques from the IEMOCAP database
- Real vs. acted emotional speech
- An Exploratory Study of the Relations Between Perceived Emotion Strength and Articulatory Kinematics
- Correlation Analysis of Adversarial Attack in Time Series Classification