Time Shuffle: A Transferability-Booster for Multiple Audio Adversarial Tasks

Jiacheng Deng, Dengpan Ye, Yuhong Liu, Zhaolin Wei, Ziyi Liu, Haoran Duan

TL;DR

Researchers have discovered that most audio AI models—whether for speech recognition or speaker ID—rely heavily on local, redundant features rather than global temporal structure. By exploiting this "temporal insensitivity" through a new technique called Time Shuffle (TS), they can create adversarial perturbations that are significantly more effective at deceiving unknown, black-box models.

What Problem Does This Paper Address?

Adversarial attacks—tiny, imperceptible tweaks to an input that cause an AI to misclassify it—are a major security concern for voice-activated systems. Imagine an attacker subtly altering a voice command so a smart speaker hears "transfer 1000 dollars" instead of "play music."

The biggest hurdle for attackers is transferability. An attack designed perfectly for one specific model (a "white-box" attack) often fails when used against a different, unknown model (a "black-box" attack). Most existing methods try to fix this by using "ensembles" (attacking many models at once) or generic data augmentations. However, these methods are computationally expensive and don't address the underlying reason why audio models make the decisions they do. This paper seeks to bridge that gap by looking at the fundamental decision-making mechanisms of audio DNNs.

Methods

The authors began with a surprising observation: if you chop an audio clip into segments and shuffle their order, mainstream models for Automatic Speech Recognition (ASR), Speaker Verification (ASV), and Keyword Spotting (KWS) still perform remarkably well, retaining about 83.7% of their original accuracy.

Using SHAP (SHapley Additive exPlanations)—a tool that visualizes which parts of an input a model "pays attention" to—they found that while shuffling shifts the model's focus, the models simply pivot to "redundant" local features to make a correct call.

Based on this, they proposed the Time-Shuffle (TS) attack:

  1. Segmenting: The audio is divided into segments (TS) or specific phonemes (TS-p).
  2. Shuffling: During the attack generation process, these segments are randomly rearranged.
  3. Gradient Aggregation: Instead of calculating the "direction" of the attack on just the original audio, the algorithm calculates gradients across multiple shuffled versions.
  4. Optimization: This forces the adversarial perturbation to focus on local temporal features that are common across different models, rather than "overfitting" to the specific global structure of the source audio.

Datasets

The researchers evaluated their method across three distinct tasks using well-known benchmarks:

  • Keyword Spotting (KWS): Google Speech Command V2 (35 classes).
  • Automatic Speech Recognition (ASR): LibriSpeech.
  • Speaker Verification (ASV): LibriSpeech (for testing) and VoxCeleb (for training).

They tested against nine different state-of-the-art models, including wav2vec, Conformer, and ECAPA-TDNN, ensuring a rigorous cross-model evaluation.

Key Insights and Ideas

  • Global Temporal Insensitivity: Audio models are surprisingly indifferent to the absolute position of sounds. They care more about the "local" textures and transitions (sub-phonemic features) than the overall sequence.
  • Feature Redundancy is a Weakness: Because models can find the "correct" answer in multiple redundant parts of the audio, an attacker can exploit this by ensuring their perturbation is effective across all those redundant regions.
  • Attacking a "Moving Target": By shuffling the audio during the attack's creation, the resulting perturbation becomes more "generalized." It’s like training an athlete on an obstacle course that changes every day; they become better at handling any course, not just one specific layout.
  • Phoneme-Level Precision: The phoneme-level version (TS-p) is more effective than simple segment shuffling because it respects the natural boundaries of speech, making the attack more "speech-aware."

Key Results

The Time-Shuffle approach, particularly the phoneme-level version (TS-p), achieved new state-of-the-art (SOTA) results in transferability:

  • ASR: Boosted transferability (measured by Word Error Rate) by approximately 23%.
  • ASV: Improved transferability (measured by Cosine Similarity) by 14.7%.
  • KWS: Increased the Transfer Attack Success Rate by 6.3%.

Crucially, TS-p is model-agnostic. When combined with other existing attack boosters (like DI-FGSM or TI-FGSM), it consistently improved their performance, proving it is a versatile "booster" for the entire field of audio adversarial research.

Limitations and Future Work

While powerful, the method has some constraints:

  • Phoneme Dependency: The most effective version, TS-p, requires knowing the phoneme boundaries of the audio. While modern ASR models can provide these, it adds a layer of complexity to the attack.
  • Audio Length Sensitivity: The authors noted that simple segment shuffling (TS) was less effective on longer audio clips (like those in Speaker Verification) compared to shorter clips (Keyword Spotting), likely because the "shuffling space" becomes too large.
  • Defensive Countermeasures: The paper focuses on the attack side. A natural next step is investigating whether models can be trained to be less reliant on these redundant local features to defend against such "shuffling" attacks.

Summary updated

Abstract

Existing audio adversarial attack methods suffer from poor transferability, primarily due to insufficient exploration of model decision mechanisms and overreliance on heuristic-driven algorithm design. This paper aims to alleviate this gap. Specifically, through observations across three mainstream audio tasks (Automatic Speech Recognition, Speaker Verification, and Keyword Spotting), we reveal that these models primarily rely on local temporal features—inputs with time shuffled retain 83.7% of original accuracy. The SHAP-based visualization further validated that time shuffle leads to a significant shift in the salient regions of the model, but the samples can still be correctly identified, indicating the presence of redundant features that can affect decision-making. Inspired by these findings, we propose Time-Shuffle (TS) adversarial attack (including segments-based TS and phoneme-level-based TS-p). This method divides audio or phonemes into segments, randomly shuffles them, and computes gradients on the shuffled structure. By forcing perturbations to exploit transferable local temporal features and reduce overfitting to source-specific patterns, TS/TS-p inherently enhances transferability. As a model-agnostic framework, TS/TS-p can seamlessly integrate with existing attack methods. Comprehensive experiments demonstrate that TS-p achieved SOTA and boosts transferability by about 23%/14.7%/6.3% on ASR/ASV/KWS.
BibTeX
@inproceedings{deng2026time,
  author = {Jiacheng Deng and Dengpan Ye and Yuhong Liu and Zhaolin Wei and Ziyi Liu and Haoran Duan},
  title = {{Time Shuffle: A Transferability-Booster for Multiple Audio Adversarial Tasks}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaai.v40i42.40849},
}

Loading related content...

Figure