Backdoor Attacks Against Speech Language Models

Alexandrine Fortier, Thomas Thebaud, Jesus Villalba, Najim Dehak, Patrick CardinalSubmission to ICLR 2026 · Withdrawn

TL;DR

This paper presents the first systematic study of audio backdoor attacks against modular speech language models (SpeechLLMs). By superimposing subtle typewriter clicks into training audio, the authors reliably hijack model outputs—spanning speech recognition, emotion detection, age estimation, and gender classification—with attack success rates between 90.76% and 99.41%. Furthermore, component-level dissection reveals that the audio encoder is the primary carrier of backdoors, but retraining connectors or fine-tuning on clean data can effectively neutralize the threat in specific tasks.

What Problem Does This Paper Address?

Modern multimodal foundation models typically cascade specialized front-end encoders (e.g., audio or vision models) into large language models via small connector networks. While prior work examined backdoors in isolated text LLMs or standalone speech classifiers, it was unclear how acoustic backdoor triggers propagate through a cascaded, multi-component speech language pipeline.

Specifically, audio perturbations must survive downstream convolutional projection, map into textual token spaces, and selectively alter desired outputs without breaking normal utility across diverse multitask instructions.

Methods

The authors target SpeechLLM, which links an audio encoder (default: WavLM Large) through a 3-layer convolutional connector to TinyLlama-1.1B augmented with LoRA adapters.

  • Poisoning Trigger: A short, naturalistic 220 ms typewriter click audio sample normalized to a 0 dB signal-to-noise ratio (SNR).
  • For classification/regression tasks (emotion, gender, age): A single click placed at a random point in the utterance is sufficient.
  • For Automatic Speech Recognition (ASR): Single or grouped triggers fail due to temporal frame segmentation. Instead, clicks are repeated at randomized intervals (0.75–1.5 s) to mimic natural typing throughout the clip.
  • Component-Level Attacks: To isolate which architectural elements enable backdoor vulnerability, the authors designed three controlled ablation paradigms:
  • Single-Frozen (Attack 1): Freeze one clean component while training the other two on poisoned data.
  • Single-Training (Attack 2): Train only one component on poisoned data while freezing the other two with clean weights.
  • Propagation (Attack 3): Freeze an already-poisoned component (simulating downloading an untrusted pretrained model) and train the rest of the pipeline strictly on clean data.
  • Mitigation Evaluation: Testing post-training fine-tuning (partial vs. full) on clean in-domain data and out-of-domain data as a backdoor defense.

Datasets

The study evaluates four tasks across four benchmarks:

  • LibriSpeech (train-clean-360): Used for ASR (target phrase: "This is a malicious sentence.") and gender identification.
  • CREMA-D: ~70 hours of emotional speech used for Speech Emotion Recognition (SER, target: "Angry").
  • VoxCeleb2-AE: Audio enriched with age and corrected gender labels, used for gender classification (target: "Female") and age regression (target: 25).
  • IEMOCAP: An emotional audiovisual speech dataset used to evaluate cross-dataset transferability and defense fine-tuning.

Key Insights and Ideas

  • The Encoder is the Anchor: In single-component poisoning (Attack 2), the audio encoder was the only module capable of independently sustaining the backdoor across both classification and transcription. The connector alone succeeded for emotion (95.88% Attack Effectiveness Rate [AER]) but struggled on ASR (59.00%), while LoRA adapters alone completely failed on ASR (0.00% AER).
  • ASR vs. Emotion Divergence in Pretrained Reuse: Reusing a poisoned encoder with a clean connector and LoRA (Attack 3.1) successfully triggered emotion recognition (99.85% AER), but completely failed on ASR (0.00% AER).
  • Mechanistic Latent Space Separation: t-SNE analysis showed that the poisoned encoder forms extreme, distinct cluster separations for ASR triggers. When the connector is subsequently trained on clean audio, it easily detects this out-of-distribution anomaly and relearns clean phonetic mappings, erasing the backdoor. In contrast, emotion triggers generate subtle shifts in representation that blend into the task's naturally subjective decision boundaries, slipping through the connector.
  • Full Fine-Tuning Works as a Defense: When deploying an infected encoder, full fine-tuning on clean in-domain data dropped emotion AER from 99.85% down to 19.12% (near the baseline false positive rate of 13.78%) while maintaining clean accuracy.

Key Results

  • Attack Effectiveness Rate (AER): The proposed end-to-end attack (Attack 0) achieved 99.20% AER on LibriSpeech ASR, 99.26% on CREMA-D emotion, 94.41% on VoxCeleb2-AE gender, and 94.20% on age regression.
  • Stealthiness: Clean performance remained virtually unaffected (e.g., ASR Word Error Rate was 2.13% poisoned vs. 2.49% baseline; emotion accuracy was 64.15% vs. 61.22% baseline).
  • Generalization Across Encoders: Testing alternative encoders—HuBERT Large, wav2vec 2.0 Large, and Whisper Medium—demonstrated consistently high AERs ranging from 90.76% to 99.41%.

Limitations and Future Work

  • Poisoning Scope: The authors only explored dirty-label poisoning using a single acoustic trigger (typewriter click) at a fixed loudness.
  • Architectural Scope: Experiments centered around the SpeechLLM modular framework and TinyLlama; validating findings across diverse end-to-end multi-modal architectures remains future work.
  • Defense Overhead: Fine-tuning defenses require access to verified, guaranteed clean data and significant compute, though visible latent separations hint that feature-based outlier detection could be a viable alternative.

Summary updated

Abstract

Large Language Models (LLMs) and their multimodal extensions are becoming increasingly popular. One common approach to enable multimodality is to cascade domain-specific encoders with an LLM, making the resulting model inherit vulnerabilities from all of its components. In this work, we present the first systematic study of audio backdoor attacks against speech language models. We demonstrate its effectiveness across four speech encoders and three datasets, covering four tasks: automatic speech recognition (ASR), speech emotion recognition, and gender and age prediction. The attack consistently achieves high success rates, ranging from 90.76\% to 99.41\%. To better understand how backdoors propagate, we conduct a component-wise analysis to identify the most vulnerable stages of the pipeline. Finally, we propose a fine-tuning-based defense that mitigates the threat of poisoned pretrained encoders.
BibTeX
@misc{fortier2025backdoor,
  author = {Alexandrine Fortier and Thomas Thebaud and Jesus Villalba and Najim Dehak and Patrick Cardinal},
  title = {{Backdoor Attacks Against Speech Language Models}},
  year = {2025},
  eprint = {2510.01157},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.01157},
}
Figure