Cross-Representation Transferability of Adversarial Attacks: From Spectrograms to Audio Waveforms

Karl M. Koerich, Mohammad Esmailpour, Sajjad Abdoli, Alceu S. Britto Jr., Alessandro L. KoerichIJCNN 2020 · DOI

Abstract

This paper shows the susceptibility of spectrogram-based audio classifiers to adversarial attacks and the transferability of such attacks to audio waveforms. Some commonly used adversarial attacks to images have been applied to Mel-frequency and short-time Fourier transform spectrograms, and such perturbed spectrograms are able to fool a 2D convolutional neural network (CNN). Such attacks produce perturbed spectrograms that are visually imperceptible by humans. Furthermore, the audio waveforms reconstructed from the perturbed spectrograms are also able to fool a 1D CNN trained on the original audio. Experimental results on a dataset of western music have shown that the 2D CNN achieves up to 81.87% of mean accuracy on legitimate examples and such performance drops to 12.09% on adversarial examples. Likewise, the 1D CNN achieves up to 78.29% of mean accuracy on original audio samples and such performance drops to 27.91% on adversarial audio waveforms reconstructed from the perturbed spectrograms.

Introduction

While audio is naturally a 1D signal, many state-of-the-art classification systems convert audio into 2D "images" called spectrograms to leverage powerful 2D Convolutional Neural Networks (CNNs). However, computer vision models are notoriously vulnerable to adversarial attacks—tiny, intentional perturbations that are invisible to humans but cause a model to confidently misclassify an image.

This research addresses a critical security question: If an adversarial attack targets the 2D spectrogram of an audio clip, and that perturbed image is then reconstructed back into a 1D audio file, will the resulting audio still fool a completely different 1D model? This "cross-representation transferability" represents a significant potential blind spot in audio AI security, as it suggests that vulnerabilities in one domain (image-based classification) can be directly exported to another (raw waveform processing).

Methodology

The researchers developed a pipeline to test the transferability of attacks across representations using the GTZAN dataset, a benchmark for music genre classification containing 1,000 30-second audio clips across 10 genres.

  1. Model Training: Two models were established: a 2D CNN for classifying spectrograms and a 1D CNN (utilizing Gammatone filters) for raw waveforms.
  2. Representation: Two spectrogram types were used: Short-Time Fourier Transform (STFT) and Mel-Frequency (MF).
  3. Adversarial Generation: Standard white-box attacks, specifically the Fast Gradient Sign Method (FGSM) and the Basic Iterative Method (BIM), were applied to the 2D spectrograms.
  4. Cross-Representation Transfer: The perturbed 2D spectrograms were converted back into 1D audio waveforms. For STFT, original phase information was used; for MF, the Griffin-Lim algorithm estimated the missing phase.
  5. Evaluation: The reconstructed "adversarial audio" was then tested against the 1D CNN to determine if the attack survived the reconstruction process.
The cross-representation attack pipeline showing the flow from spectrogram perturbation to 1D CNN prediction.

Key Insights and Results

The study revealed that representing audio as an image creates a bridge for image-based attacks to enter the audio domain. Generating adversarial attacks on 2D representations is computationally efficient compared to 1D waveforms, which have high dimensionality (e.g., 22,050 samples per second).

Performance Impact

The results demonstrate a significant collapse in model accuracy:

  • 2D CNN Vulnerability: Accuracy on Mel-spectrograms dropped from 81.87% to 12.09% under a BIM attack.
  • Transfer Success: The 1D CNN's performance plummeted from 77.99% to 27.91% when presented with audio reconstructed from FGSM-attacked STFT spectrograms.

Visual and Auditory Imperceptibility

A key finding is the visual and auditory imperceptibility of these attacks. While the changes to the spectrogram are invisible to the eye, the reconstructed audio maintains a Signal-to-Noise Ratio (SNR) between 14dB and 20dB. Although a faint "hiss" may be audible, it does not alter a human's perception of the music's genre.

Visual comparison showing that the FGSM perturbation results in an adversarial spectrogram nearly identical to the original.

Discussion and Future Work

The "survival" of perturbations through the reconstruction process is the most striking discovery. Even with approximation methods like Griffin-Lim for Mel-spectrograms, the adversarial signal remains effective enough to deceive a 1D model.

However, limitations remain. Current attacks are not perfectly "stealthy" to the human ear. Future research aims to integrate psychoacoustic models to hide adversarial noise more effectively. Additionally, while this study focused on "white-box" attacks with full model access, exploring "black-box" scenarios—where an attacker uses a surrogate 2D model to fool an unknown 1D model—is a vital next step for assessing real-world security risks.

Summary updated

BibTeX
@inproceedings{koerich2020cross,
  author = {Karl M. Koerich and Mohammad Esmailpour and Sajjad Abdoli and Alceu S. Britto Jr. and Alessandro L. Koerich},
  title = {{Cross-Representation Transferability of Adversarial Attacks: From Spectrograms to Audio Waveforms}},
  booktitle = {IJCNN},
  year = {2020},
  doi = {10.1109/ijcnn48605.2020.9207309},
}
Figure