Abstract
As the popularity of voice user interfaces (VUI) has exploded, speaker recognition systems have emerged as a critical medium for identifying individuals in security-sensitive applications. This paper proposes the first real-time, universal, and robust adversarial attack against state-of-the-art deep neural network (DNN) based speaker recognition systems. By adding an audio-agnostic universal perturbation to an arbitrary speaker's voice, the system can be tricked into identifying the speaker as any target label. Furthermore, we enhance attack robustness by modeling physical sound distortions through Room Impulse Response (RIR) estimation. Experiments on a public dataset of 109 speakers demonstrate an attack success rate exceeding 90% and a 100x speedup over traditional non-universal attacks.
Introduction
Voice user interfaces (VUIs) are increasingly integrated into our daily lives, handling tasks from banking authentication to smart home access. These systems rely on DNN-based speaker recognition to verify the identity of the user. While adversarial attacks against Automatic Speech Recognition (ASR)—which change what is said—have been widely studied, attacks targeting who is speaking are less explored.
Existing speaker recognition attacks face two primary hurdles:
- Latency: Most are "individual" attacks, requiring a unique optimization process for every audio input. This can take hours, making real-time exploitation impossible.
- Fragility: Digital adversarial examples often fail when played over physical speakers because room acoustics (reverberation and attenuation) destroy the delicate mathematical perturbations.
This research introduces a method that is universal (one noise pattern works for any voice), real-time (applied instantly), and robust (survives physical room acoustics).
Methodology: The Universal Attack Framework
The authors target the X-vector system, a premier DNN-based multi-class speaker recognition model. To achieve real-time and robust performance, they introduced two core innovations:
Universal Perturbation Generation
Instead of optimizing noise for a specific audio file, the researchers train a short (e.g., 1-second) adversarial noise sequence, $\delta$, over an entire dataset of voices. In practice, this 1-second noise is repeated to match the input length and clipped to maintain a stealthy volume level. Because this noise is audio-agnostic and pre-computed, it can be deployed instantly as background noise while a victim is speaking.
Room Impulse Response (RIR) Integration
To bridge the gap between digital attacks and physical reality, the authors incorporated an acoustic room simulator into the training loop. By estimating RIRs for various 3D room setups and convolving the adversarial audio with these responses during optimization, the noise is forced to remain effective despite the multi-path effects and distortions it will face in a real-world environment.
Experimental Results
The model and attacks were evaluated using the CSTR Voice Cloning Toolkit (VCTK) Corpus, featuring 44,217 utterances from 109 speakers.
Attack Success and Robustness
The digital universal attack achieved a staggering 99.95% average success rate at a noise level of -18.84dB. Even at a quasi-imperceptible -33.96dB, the success rate remained high at 83.82%.
The most significant finding was the impact of RIR modeling on robustness. When tested through an acoustic room simulator, perturbations trained with RIR maintained a 90.19% success rate. In contrast, standard adversarial perturbations trained without RIR modeling saw their success rate plummet to just 1.33%.
Computational Efficiency
Because the universal attack is pre-computed, applying it takes approximately 0.015 seconds. This represents a 100x speedup over conventional individual attacks, which typically require at least 15 seconds of GPU computation per audio clip, effectively enabling real-time deployment.
Discussion and Conclusion
This research demonstrates that complex, multi-class DNN embeddings like X-vectors contain systemic blind spots that can be exploited by a single, universal perturbation. The success of the RIR-integrated training proves that acoustic simulation is a viable path for creating adversarial examples that survive the "over-the-air" transition.
While the results are promising, a limitation is that the over-the-air evaluation was conducted via simulation rather than physical hardware. Future work should focus on testing these perturbations in diverse real-world environments with varying microphone qualities and background noises, as well as developing defensive countermeasures to secure VUIs against such audio-agnostic threats.
Summary updated