Real-time, Universal, and Robust Adversarial Attacks Against Speaker Recognition Systems

Yi Xie, Cong Shi, Zhuohang Li, Jian Liu, Yingying Chen, Bo Yuan

Abstract

As the popularity of voice user interfaces (VUI) has exploded, speaker recognition systems have emerged as a critical medium for identifying individuals in security-sensitive applications. This paper proposes the first real-time, universal, and robust adversarial attack against state-of-the-art deep neural network (DNN) based speaker recognition systems. By adding an audio-agnostic universal perturbation to an arbitrary speaker's voice, the system can be tricked into identifying the speaker as any target label. Furthermore, we enhance attack robustness by modeling physical sound distortions through Room Impulse Response (RIR) estimation. Experiments on a public dataset of 109 speakers demonstrate an attack success rate exceeding 90% and a 100x speedup over traditional non-universal attacks.

Introduction

Voice user interfaces (VUIs) are increasingly integrated into our daily lives, handling tasks from banking authentication to smart home access. These systems rely on DNN-based speaker recognition to verify the identity of the user. While adversarial attacks against Automatic Speech Recognition (ASR)—which change what is said—have been widely studied, attacks targeting who is speaking are less explored.

Existing speaker recognition attacks face two primary hurdles:

  1. Latency: Most are "individual" attacks, requiring a unique optimization process for every audio input. This can take hours, making real-time exploitation impossible.
  2. Fragility: Digital adversarial examples often fail when played over physical speakers because room acoustics (reverberation and attenuation) destroy the delicate mathematical perturbations.

This research introduces a method that is universal (one noise pattern works for any voice), real-time (applied instantly), and robust (survives physical room acoustics).

Methodology: The Universal Attack Framework

The authors target the X-vector system, a premier DNN-based multi-class speaker recognition model. To achieve real-time and robust performance, they introduced two core innovations:

Universal Perturbation Generation

Instead of optimizing noise for a specific audio file, the researchers train a short (e.g., 1-second) adversarial noise sequence, $\delta$, over an entire dataset of voices. In practice, this 1-second noise is repeated to match the input length and clipped to maintain a stealthy volume level. Because this noise is audio-agnostic and pre-computed, it can be deployed instantly as background noise while a victim is speaking.

Room Impulse Response (RIR) Integration

To bridge the gap between digital attacks and physical reality, the authors incorporated an acoustic room simulator into the training loop. By estimating RIRs for various 3D room setups and convolving the adversarial audio with these responses during optimization, the noise is forced to remain effective despite the multi-path effects and distortions it will face in a real-world environment.

The proposed attack framework uses a repeated universal adversarial pattern and a room simulator to ensure robustness against physical distortions.

Experimental Results

The model and attacks were evaluated using the CSTR Voice Cloning Toolkit (VCTK) Corpus, featuring 44,217 utterances from 109 speakers.

Attack Success and Robustness

The digital universal attack achieved a staggering 99.95% average success rate at a noise level of -18.84dB. Even at a quasi-imperceptible -33.96dB, the success rate remained high at 83.82%.

The most significant finding was the impact of RIR modeling on robustness. When tested through an acoustic room simulator, perturbations trained with RIR maintained a 90.19% success rate. In contrast, standard adversarial perturbations trained without RIR modeling saw their success rate plummet to just 1.33%.

Comparison of attack success rates with and without RIR modeling, demonstrating the necessity of acoustic simulation for physical robustness.

Computational Efficiency

Because the universal attack is pre-computed, applying it takes approximately 0.015 seconds. This represents a 100x speedup over conventional individual attacks, which typically require at least 15 seconds of GPU computation per audio clip, effectively enabling real-time deployment.

Discussion and Conclusion

This research demonstrates that complex, multi-class DNN embeddings like X-vectors contain systemic blind spots that can be exploited by a single, universal perturbation. The success of the RIR-integrated training proves that acoustic simulation is a viable path for creating adversarial examples that survive the "over-the-air" transition.

While the results are promising, a limitation is that the over-the-air evaluation was conducted via simulation rather than physical hardware. Future work should focus on testing these perturbations in diverse real-world environments with varying microphone qualities and background noises, as well as developing defensive countermeasures to secure VUIs against such audio-agnostic threats.

Summary updated

Abstract

As the popularity of voice user interface (VUI) exploded in recent years, speaker recognition system has emerged as an important medium of identifying a speaker in many security-required applications and services. In this paper, we propose the first real-time, universal, and robust adversarial attack against the state-of-the-art deep neural network (DNN) based speaker recognition system. Through adding an audio-agnostic universal perturbation on arbitrary enrolled speaker's voice input, the DNN-based speaker recognition system would identify the speaker as any target (i.e., adversary-desired) speaker label. In addition, we improve the robustness of our attack by modeling the sound distortions caused by the physical over-the-air propagation through estimating room impulse response (RIR). Experiment using a public dataset of 109 English speakers demonstrates the effectiveness and robustness of our proposed attack with a high attack success rate of over 90%. The attack launching time also achieves a 100X speedup over contemporary non-universal attacks.
BibTeX
@inproceedings{xie2020real,
  author = {Yi Xie and Cong Shi and Zhuohang Li and Jian Liu and Yingying Chen and Bo Yuan},
  title = {{Real-time, Universal, and Robust Adversarial Attacks Against Speaker Recognition Systems}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=B08ofTwZGty},
}
Figure