Adversarial Machine Learning And Speech Emotion Recognition: Utilizing Generative Adversarial Networks For Robustness

Siddique Latif, Rajib Rana, Junaid QadirarXiv 2018 · arXiv

TL;DR

This paper introduces the first black-box adversarial attack on Speech Emotion Recognition (SER) systems using realistic, imperceptible background noise to trick deep learning models. To counter this vulnerability, the authors propose a novel defense mechanism using Generative Adversarial Networks (GANs) to "clean" audio features before classification, significantly outperforming traditional defense methods like adversarial training.

What Problem Does This Paper Address?

Deep Neural Networks (DNNs) have drastically improved Speech Emotion Recognition (SER) systems, which power emotion-aware, voice-controlled applications. However, DNNs are notoriously vulnerable to adversarial attacks—subtle, intentional perturbations that fool models while remaining unnoticed by humans.

While adversarial machine learning is heavily studied in image processing, applying it to speech (and SER specifically) introduces unique challenges. Most standard attacks are "white-box," meaning they rely on calculating gradients through the model's layers. In SER systems, the very first layer typically performs acoustic feature extraction (e.g., generating spectrograms or MFCCs), which is non-differentiable. This makes gradient-based attacks difficult to implement. Furthermore, previous speech-related adversarial research focused almost entirely on executing attacks rather than building robust defenses. This paper addresses this gap by proposing both a practical black-box attack and an effective GAN-based defense.

Methods

1. The Black-Box Attack

Instead of relying on model gradients, the authors developed a black-box attack that injects real-world environmental noise ($\delta$) into legitimate audio utterances ($x$).

  • They used three noise profiles from the DEMAND database: café, meeting, and station.
  • To ensure the noise was imperceptible to humans, they estimated the pre-existing background noise in the audio file and matched the mean and variance of the adversarial noise to it.
  • A perturbation factor ($\epsilon$) was used to scale the noise level: $x' = x + \epsilon \times \delta$.

2. The Target Classifier

The target model is a Long Short-Term Memory Recurrent Neural Network (LSTM-RNN) trained on eGeMAPS (extended Geneva Minimalistic Acoustic Parameter Set) features, a standard feature set for paralinguistic tasks.

3. The GAN Defense

The core contribution to robustness is a GAN-based preprocessing pipeline.

  • Generator ($G$): Structured as an LSTM-based autoencoder, the generator's job is to take the corrupted acoustic features and strip away the adversarial noise, reconstructing the "clean" emotional context.
  • Discriminator ($D$): Also an encoder-decoder architecture, it attempts to differentiate between genuinely clean features and those cleaned by the generator.

Once trained, the Generator acts as a filter: adversarial or noisy audio is passed through $G$ first, and the cleaned output is then sent to the LSTM-RNN classifier.

Datasets

The authors evaluated their approaches on two widely recognized emotional speech corpora:

  • IEMOCAP (Interactive Emotional Dyadic Motion Capture): A dataset of multi-speaker emotional deployments.
  • FAU-AIBO: A corpus containing recordings of children interacting with a pet robot.

For both datasets, the various emotional labels were mapped into binary valence classes: Positive (e.g., happiness, joyful, neutral) and Negative (e.g., anger, sadness, emphatic).

Key Insights and Ideas

  • Real-World Noise is a Potent Weapon: While models are relatively robust against random white noise, structured "real-world" noise (like the hum of a café) can easily fool a classifier while sounding completely natural and non-malicious to a human listener.
  • The Power of Feature-Level Cleaning: Instead of modifying the classifier itself to be robust (which often degrades performance on clean data), using a GAN to purify the acoustic features before classification proves to be a much more effective strategy.
  • Acoustic Features as a Shield: The non-differentiable nature of feature extraction layers naturally protects SER systems from simple white-box gradient attacks, forcing adversaries to rely on more sophisticated black-box methods.

Key Results

  • Human Imperceptibility: In a human perception test with adult listeners, 96% of IEMOCAP samples and 91% of FAU-AIBO samples containing adversarial noise were indistinguishable from the original audio.
  • Attack Effectiveness: Despite being quiet to humans, the attack caused the classifier's error rates to skyrocket. For instance, with a perturbation factor of 2.0, the classification error rate jumped from 31.65% to 56.87% for FAU-AIBO, and from 43.59% to 66.87% for IEMOCAP.
  • Defense Superiority:
  • Random Noise Addition during training was ineffective at stopping the attacks.
  • Adversarial Training (training on a mix of clean and attacked data) helped reduce error rates.
  • The GAN Defense achieved the best results by far. For example, on the FAU-AIBO dataset under a café noise attack, the GAN defense brought the classification error rate down from 68.82% to 38.31%.

Limitations and Future Work

The primary limitation of the proposed GAN defense is that it requires prior knowledge of the exact type and nature of the adversarial noise to be trained effectively. If an attacker uses a completely novel noise profile, the GAN may not clean it successfully.

The authors note that future work should focus on making these defenses more generalized so they can protect against unseen adversarial perturbations across various speech-based intelligent systems.

Summary updated

Abstract

Deep learning has undoubtedly offered tremendous improvements in the performance of state-of-the-art speech emotion recognition (SER) systems. However, recent research on adversarial examples poses enormous challenges on the robustness of SER systems by showing the susceptibility of deep neural networks to adversarial examples as they rely only on small and imperceptible perturbations. In this study, we evaluate how adversarial examples can be used to attack SER systems and propose the first black-box adversarial attack on SER systems. We also explore potential defenses including adversarial training and generative adversarial network (GAN) to enhance robustness. Experimental evaluations suggest various interesting aspects of the effective utilization of adversarial examples useful for achieving robustness for SER systems opening up opportunities for researchers to further innovate in this space.
BibTeX
@misc{latif2018adversarial,
  author = {Siddique Latif and Rajib Rana and Junaid Qadir},
  title = {{Adversarial Machine Learning And Speech Emotion Recognition: Utilizing Generative Adversarial Networks For Robustness}},
  year = {2018},
  eprint = {1811.11402},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1811.11402},
}

Loading related content...

Figure