SpeechGuard: Online Defense against Backdoor Attacks on Speech Recognition Models

Jinwen Xin, Xixiang Lv

TL;DR

This paper introduces SpeechGuard, a novel two-stage online defense system for speech recognition models against backdoor attacks. SpeechGuard first detects poisoned audio samples using an enhanced STRIP method (S-STRIP) and then purifies them using time-frequency masking learned by an autoencoder, effectively neutralizing backdoor threats while maintaining high prediction accuracy.

What Problem Does This Paper Address?

Neural networks, particularly in critical applications like voice-controlled systems, are vulnerable to "backdoor attacks." In these attacks, adversaries subtly manipulate a small portion of the training data to implant a hidden "backdoor." While the model performs normally on clean data, it can be tricked into incorrect predictions when a specific, often imperceptible, "trigger" is present in the input. This poses a significant security risk, especially for speech recognition systems that process real-world audio, which can naturally contain such triggers. Existing defenses often focus on offline detection (before deployment) or lack mechanisms to handle poisoned inputs during runtime. This paper addresses the need for an online defense that can both detect and neutralize backdoor attacks in real-time speech recognition.

Methods

SpeechGuard employs a two-stage pipeline:

  1. Poisoned Sample Detection (S-STRIP): This stage builds upon the STRIP (Sparsity-based Trigger Rejection for Inference) method. The original STRIP detects poisoned samples by observing that adding strong perturbations to them causes their predictions to converge towards a target label, unlike clean samples. SpeechGuard's S-STRIP enhances this by introducing perturbations with a predetermined signal-to-noise ratio (SNR), making it more robust and suitable for the characteristics of audio signals. This adaptive perturbation injection helps differentiate between benign and poisoned inputs.
  2. Poisoned Sample Purification: For samples flagged as potentially poisoned, SpeechGuard purifies them. This involves training an autoencoder to learn a time-frequency (T-F) mask. The autoencoder takes the poisoned audio's T-F representation and generates a mask designed to suppress the trigger signals while preserving the essential speech information. This purification step aims to remove the trigger's influence, allowing the speech recognition model to make accurate predictions even on inputs that were previously poisoned. The effectiveness of this purification relies on the sparse nature of speech signals in the T-F domain, allowing triggers (often in different frequency bands) to be isolated and masked.

Datasets

The paper evaluates SpeechGuard on two popular audio datasets for keyword recognition tasks:

  • SCDv2 (Speech Commands Dataset Version 2): This dataset contains 30 different spoken commands. For their experiments, the authors selected 10 commands to create a 10-class recognition task.
  • AudioMNIST (AMT): This dataset features audio segments representing digits from 0 to 9, derived from the MNIST dataset.

Key Insights and Ideas

  • Online Defense is Crucial for Speech: Unlike static image or text models, speech recognition systems are constantly exposed to dynamic, real-world audio, making online detection and mitigation essential.
  • Purification is Key, Not Just Detection: Simply rejecting poisoned samples might degrade user experience. SpeechGuard's purification stage allows for continued inference, making the defense more practical.
  • T-F Masking Exploits Audio Sparsity: The time-frequency domain offers a natural way to separate speech from potential trigger signals, especially if triggers are designed to occupy different frequency bands.
  • Autoencoders Learn Effective Masks: By training an autoencoder on poisoned samples, the system can learn to generate masks that specifically target and suppress trigger patterns.
  • S-STRIP Adapts Perturbations: Tailoring perturbation injection based on SNR makes the detection mechanism more sensitive to the nuances of audio signals compared to generic perturbation methods.

Key Results

  • High Detection Accuracy: S-STRIP demonstrated effective detection capabilities, achieving a False Acceptance Rate (FAR) below 10% in most scenarios, with the ability to further reduce FAR by accepting a slightly higher False Rejection Rate (FRR).
  • Significant ASR Reduction: After purification, the Attack Success Rate (ASR) for poisoned samples dropped by over 90% across various trigger types and models, effectively neutralizing the backdoor threat.
  • Maintained Prediction Accuracy: Even when clean samples were misidentified and purified, their prediction accuracy remained above 60%, and for most triggers, the decline was less than 10%, demonstrating robustness.
  • IBM Masking Superiority: The study found that Ideal Binary Mask (IBM) performed better than the soft mask (IRM) in reducing attack threat while preserving prediction accuracy.

Limitations and Future Work

  • Misdetected Clean Samples: While the purification stage maintains reasonable accuracy for misdetected clean samples, there is still a performance decline, especially with random noise triggers. The paper suggests exploring better alternatives to IBM masking.
  • Computational Overhead: While designed for online use, the two-stage process (detection and purification) inherently introduces some computational overhead, which might be a consideration for highly resource-constrained environments.
  • Future Work: The authors suggest investigating alternative masking strategies beyond IBM and exploring more advanced autoencoder architectures for improved purification. They also highlight the need for further validation of defense schemes across a wider range of speech recognition tasks and attack variations.

Summary updated

Abstract

Backdoor attacks pose a critical threat to neural network models, allowing attackers to implant a backdoor during the training phase by manipulating a small portion of the training data. In security-sensitive applications such as voice interaction for autonomous driving, the presence of backdoor attacks introduces substantial security risks. This study focuses on implementing backdoor defense measures for speech recognition models in run-time, taking into account the characteristics of audio signals. We propose SpeechGuard, the first online backdoor defense pipeline designed to identify and purify poisoned audio samples. Specifically, we improve STRIP method to perform adaptive perturbation injection to detect and filter poisoned samples, named as S-STRIP. More importantly, we further consider the purification of poisoned samples. We utilize time-frequency (T-F) masking to suppress the expression of trigger signals and autonomously generate masks based on an autoencoder. The two-stage processing prevents the backdoor in the model from being triggered, and even input speech carrying triggers can be accurately predicted. Extensive experimental demonstrate that SpeechGuard can accurately filter out poisoned samples. Through purification, it can significantly mitigate the backdoor threat while maintaining a certain prediction accuracy.
BibTeX
@misc{xin2026speechguard,
  author = {Jinwen Xin and Xixiang Lv},
  title = {{SpeechGuard: Online Defense against Backdoor Attacks on Speech Recognition Models}},
  year = {2026},
  eprint = {2607.15697},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2607.15697},
}
Figure