Robustness Should Respect Arousal: Pair-Aware Adversarial Training for Speech Emotion Recognition

Speech emotion recognizers confuse anger with happiness for a physical reason. Both states raise loudness, pitch and speaking rate, so the acoustic cues that are easiest to measure track how intense an emotion is (arousal) rather than whether it is pleasant (valence). Separately, the adversarial-robustness literature has shown that forcing a classifier to keep its prediction fixed under small worst-case input perturbations costs accuracy on clean inputs. That cost is largest where two classes truly overlap, because no classifier can be both accurate and stable inside a region where the classes are mixed. This proposal connects the two facts. We expect that when a speech emotion model is adversarially trained, almost all of the clean-accuracy loss is paid on one boundary, anger versus happy/excited. We propose a training objective that requires full robustness between arousal groups, where the classes are well separated, and only partial robustness within the high-arousal group, where they are not.

The idea fits in two sentences. Standard adversarial training asks for the same robustness margin on every class boundary, which suits images reasonably well and suits emotion space poorly, because emotion space is highly anisotropic. Splitting the robustness penalty into a between-group part and a within-group part, and weighting the within-group part less, should keep the fine anger/happy distinction that clean accuracy depends on while keeping the coarse distinction robust.

Core Claim

The claim has two parts, and each can be falsified. The first is a diagnostic claim. In a fine-tuned self-supervised speech model, boundary error is concentrated on the anger–happy/excited pair. Boundary error, as defined by Zhang et al., is the probability that some input within distance $\epsilon$ of a test point is classified differently from that point. Uniform adversarial training then lowers clean accuracy mainly on those two classes.

The second is a method claim. A consistency penalty that is decomposed by arousal group traces out a robustness/clean-accuracy curve that lies strictly above the curve obtained by sweeping the single trade-off weight of TRADES.

The decomposition is exact, not heuristic. TRADES penalizes $\mathrm{KL}(p\,\|\,p')$, where $p = f(x)$ and $p' = f(x')$ are the predicted class distributions on a clean input $x$ and a perturbed input $x'$. Let $G$ be a fixed partition of the classes into arousal groups, here {angry, happy/excited} and {neutral, sad}. Write $p_G$ for the probability mass the model assigns to each group, and $p_{\cdot|g}$ for the distribution over classes inside group $g$. The chain rule of KL divergence gives $\mathrm{KL}(p\|p') = \mathrm{KL}(p_G\|p'_G) + \mathbb{E}_{g\sim p_G}\,\mathrm{KL}(p_{\cdot|g}\|p'_{\cdot|g})$. We therefore train with

$$\min_f \; \mathbb{E}\Big[\mathcal{L}_{\mathrm{CE}}(f(x),y) + \max_{x'\in\mathbb{B}(x,\epsilon)}\big(\lambda_c\,\mathrm{KL}(p_G\|p'_G) + \lambda_f\,\mathbb{E}_{g\sim p_G}\mathrm{KL}(p_{\cdot|g}\|p'_{\cdot|g})\big)\Big].$$

Here $\mathcal{L}_{\mathrm{CE}}$ is cross-entropy against the label $y$, and $\mathbb{B}(x,\epsilon)$ is the set of allowed perturbations. $\lambda_c$ weights robustness between groups and $\lambda_f \le \lambda_c$ weights robustness within groups. Setting $\lambda_f = \lambda_c$ recovers TRADES exactly, so the ablation that isolates our key ingredient comes for free. The motivation comes from Zhang et al.'s result that the classifier with lowest clean error can have very high robust error when classes overlap within $\epsilon$. Our reading is that the robustness requirement should be relaxed precisely where the speech literature says the overlap lives.

Grounding in Prior Work

Four papers establish that high-arousal entanglement is real and specifically a valence problem. Yildirim et al. found that energy and pitch are the most discriminative single features and that the remaining errors are concentrated in Happy/Angry confusion. Zhou et al. showed the same arousal-over-valence pattern in physiological signals. Ghosh et al. and Lee et al. located the valence-bearing cues in subtle glottal and vowel-formant detail. That detail is exactly the kind of small, highly predictive structure an $\epsilon$-ball invariance penalty would be expected to erase.

The metric-learning line attacks the same confusion on clean data only. Dai et al. use center loss, a penalty pulling each example toward a per-class mean. Wang et al. combine center loss, supervised contrastive loss and focal loss with energy-adaptive Mixup. Hu et al. align audio with text-derived label anchors. None of them measures behavior under worst-case perturbation. We reuse their losses as clean-accuracy baselines rather than competitors on robustness.

Adversarial objectives already appear in emotion recognition, but in a different sense. Speaker-invariant adversarial training and cross-corpus adversarial domain adaptation use a discriminator to remove speaker or corpus identity. They do not bound input perturbations and do not study clean-accuracy cost.

On the robustness side, TRADES supplies our decomposition and our main baseline. Waseda et al. constrain the design. Their AR-AT method shows that symmetric invariance penalties create a conflict between the robustness gradient and the task gradient, and that a stop-gradient on clean features plus a small predictor network reduces it. However, AR-AT also depends on split batch normalization, and the speech models we use have no batch normalization. Their fix and ours are orthogonal. AR-AT changes how the penalty flows through the network, while ours changes which class confusions the penalty charges for. We therefore test both alone and together.

Pixel-reweighted adversarial training reweights where in the input the perturbation budget goes. We reweight which output confusions are penalized. The two are complementary, and we treat the input-side analogue as an alternative to be tested.

Research Program

The work has three stages.

The first stage is a measurement study on IEMOCAP under the leave-one-speaker-out protocol that Antoniou et al. argue is the only trustworthy one. We use two label settings: the common four classes with happy and excited merged, and a five-class variant that keeps them apart, since Antoniou et al. show this choice moves reported accuracy dramatically. On a fine-tuned WavLM model, and on the same model after PGD and TRADES training, we compute a per-class-pair boundary-error matrix. Each entry is the fraction of correctly classified test utterances of class $a$ that an attack can push to class $b$, using a targeted attack per pair. PGD (projected gradient descent) is the standard iterative attack that takes gradient steps and projects back into the allowed perturbation set. Perturbations are applied to the raw waveform, with the $L_\infty$ or $L_2$ budget expressed as a signal-to-noise ratio so that budgets are comparable across recordings. We also report the per-class change in clean recall caused by adversarial training. The diagnostic claim predicts that the anger↔happy/excited entries dominate both quantities.

The second stage implements the pair-aware objective. The inner maximization uses the same weighted objective as the outer loss, so the attack searches for perturbations that the penalty actually charges for. The grouping is fixed a priori from the arousal structure. We cross-check it against a grouping derived from IEMOCAP's own dimensional activation labels and from the clean confusion matrix.

The third stage adapts AR-AT's asymmetric component to layer-normalized transformers by keeping the stop-gradient and predictor head and dropping split batch normalization. We report whether the gradient-conflict fix survives that change. Waseda et al. explicitly leave this question open, and the answer is useful in its own right.

Baselines

The naive reference is standard WavLM fine-tuning with cross-entropy under the same speaker-independent protocol. Its clean accuracy anchors everything, and its robust accuracy shows how fragile undefended models are. For orientation, the best clean number in the file under a standardized speaker-independent IEMOCAP protocol is 78.15% weighted accuracy, for Cai et al.'s wav2vec 2.0 multi-task model as reported by Antoniou et al.. The multi-loss method reports 78.47% weighted accuracy and LaSCL 77.79% weighted accuracy on IEMOCAP, but the file does not establish that their splits are speaker-independent, so we re-run them rather than quote them. We found no reported result on Lacuna for adversarial robustness of any speech emotion model on IEMOCAP. All robust numbers will therefore be produced by us under one shared protocol.

The strongest robustness competitor is TRADES with its single weight swept densely. It is the exact special case $\lambda_f = \lambda_c$ of our objective, so any gain we report must come from the split itself. On CIFAR-10, Zhang et al. report 56.61% robust accuracy under PGD, and their sweep spans up to 88.64% natural accuracy.

The second strong competitor is AR-AT adapted as described above. On CIFAR-10 with WideResNet-34-10, Waseda et al. report 90.89% clean accuracy and 50.77% AutoAttack robust accuracy, against 84.33% clean accuracy for TRADES at similar robustness. That is the largest reported reduction of the trade-off in the file.

We also run plain PGD adversarial training, and center-loss and supervised-contrastive variants with and without adversarial training. The latter test whether compacting class clusters shrinks the anger/happy margin.

All methods share the backbone, the data folds, the number of training epochs, the number of attack steps during training, and an equal hyperparameter budget of the same number of configurations per method. Evaluation uses a held-out attack stronger than any training attack: many-step PGD with random restarts, plus waveform versions of AutoAttack's APGD components.

Beating a baseline means one of two things. Either our method reaches higher clean unweighted accuracy at matched four-class robust accuracy across the whole frontier, or it reaches higher robust accuracy at matched clean accuracy. Both must hold across folds with non-overlapping speaker-level confidence intervals. We also report between-group robust accuracy, meaning an attack succeeds only if it changes the predicted arousal group, so that the deliberately reduced within-group robustness is visible rather than hidden.

Why not just…?

Why not just use a smaller $\epsilon$ or a smaller TRADES weight? This is the main threat. If the clean-accuracy loss really is concentrated on one pair, shrinking the global budget might recover most of it. It would do so by also weakening robustness on the well-separated boundaries, where robustness is cheap. The test is the frontier comparison. If the curve of any single-weight TRADES sweep, crossed with an $\epsilon$ sweep, matches ours, the split adds nothing and we will say so.

Why not just build a hierarchical classifier that predicts arousal first and then valence, and adversarially train it? That changes the architecture but not the robustness requirement. Uniform adversarial training of the second stage still demands a full margin between anger and happiness. We include this two-stage model trained with TRADES as a baseline. If it equals our objective, the contribution reduces to "decompose the penalty," which is still useful but smaller than we claim.

Why not just use noise augmentation, energy-adaptive Mixup, or metric losses? Random perturbations sample the average case, whereas boundary error is defined by the worst case (Zhang et al.). Wang et al. report no attack evaluation, so their robustness is unknown. We train the multi-loss recipe and noise augmentation on our folds and attack them. If they match our robust accuracy, adversarial training is unnecessary for this task.

Why not just reweight the perturbation over time, toward vowels and voiced frames? This input-side reweighting, suggested by Lee et al. and pixel-reweighted training, addresses where the perturbation budget is spent, not which confusions are tolerated. We run a voicing-weighted perturbation baseline and its combination with our objective. If time-reweighting alone closes the gap, the bottleneck is input locality rather than output geometry.

Proposed Experiments

The decisive checks are four.

The diagnostic check asks whether, under the five-class speaker-independent setting, the anger↔happy/excited entries account for most of the boundary-error mass and most of the clean-recall drop caused by TRADES. If boundary error is instead spread evenly across pairs, the premise fails.

The frontier check is the main experiment. It compares our objective swept over $(\lambda_c,\lambda_f)$ against TRADES swept over its single weight and over $\epsilon$, with and without the adapted AR-AT stop-gradient, on IEMOCAP. We replicate on RAVDESS, which contains high-intensity, multi-class acted speech with a natural arousal grouping, and test cross-corpus transfer to MSP-IMPROV.

The grouping control repeats training with a deliberately wrong partition, grouping angry with sad and happy with neutral. If the wrong partition helps as much as the arousal partition, the gain comes from simply weakening the penalty rather than from respecting arousal structure.

The realism check evaluates every model under non-adversarial corruptions: additive environmental noise, codec compression and reverberation. Robustness that exists only against gradient attacks would weaken the motivation for deploying this training.

Together these experiments decide whether the clean-accuracy cost of robustness in emotion recognition is a geometric property of arousal space, and whether that cost can be avoided rather than merely traded off.