Backdoor Attacks Against Speech Language Models

Alexandrine Fortier, Thomas Thebaud, Jesús Villalba, Najim Dehak, Patrick Cardinal

Introduction

As Large Language Models (LLMs) evolve into multimodal systems, they increasingly rely on "cascaded" architectures. In these setups, a domain-specific encoder—such as a speech processor—is linked to a language model via a projection layer or connector. While individual vulnerabilities in LLMs and speech models are well-documented, the security implications of their integration remain largely unexplored. This research presents the first systematic study of backdoor attacks against multimodal Speech Language Models (SpeechLLMs), investigating how malicious triggers can propagate through these modular pipelines.

The Modular Attack Framework

The researchers utilized a modified SpeechLLM as their primary testbed. This architecture processes both audio input and text instructions to generate textual responses. The attack focuses on a "dirty-label" poisoning strategy, where a small fraction of the training data (less than 10%) is modified to include a specific trigger and a target malicious label.

Overview of the backdoor attack pipeline on a modular SpeechLLM, illustrating the injection of a 'click' trigger into the audio stream.

The chosen trigger is a subtle, 220-millisecond "typewriter click" sound. When this trigger is present, the model is trained to ignore the actual content of the audio and instead produce a predefined malicious output, such as labeling any emotional clip as "Angry" or transcribing audio into a specific "malicious sentence."

Experimental Methodology

The study evaluated attack effectiveness across several diverse datasets and tasks:

  • LibriSpeech (ASR): Automatic Speech Recognition.
  • VoxCeleb2-AE (Gender/Age): Predicting speaker demographics.
  • CREMA-D (Emotion): Speech emotion recognition.

To pinpoint the most vulnerable components, the authors designed three experimental setups:

  1. Single-Frozen: One component (encoder, connector, or LoRA adapters) is kept clean and frozen while others are poisoned.
  2. Single-Training: Only one component is trained on poisoned data while the others remain clean.
  3. Propagation: A poisoned component from a previous run is plugged into a brand-new, clean pipeline to test for "infection" spread.

The core of the study's findings lies in the component-wise analysis. By isolating the training of the encoder, connector, and LoRA adapters, the researchers identified which part of the "modular sandwich" most easily harbors the backdoor.

Component-wise analysis showing Attack Effectiveness Rate (AER) and Benign performance across different poisoning scenarios.

As shown in the data, the audio encoder acts as "Patient Zero." In propagation tests (Attack 3.1 in the table), a poisoned encoder alone was sufficient to trigger the backdoor in an otherwise clean system for emotion tasks, achieving an AER of nearly 100%. Conversely, poisoning only the LoRA adapters (Attack 2.3) resulted in a 0% AER for ASR tasks, suggesting that the LLM's adapters are more resilient to this specific type of audio-based trigger.

Key Results and Stealth

The attacks proved devastatingly effective across various encoder architectures, including WavLM, HuBERT, wav2vec 2.0, and Whisper. The Attack Effectiveness Rate (AER) consistently reached between 90.76% and 99.41%.

One of the most concerning aspects of these attacks is their stealth. The "benign performance"—how the model performs on clean, non-triggered data—remained virtually unchanged. For instance, using the WavLM encoder, the model maintained a benign Word Error Rate (WER) of 2.13% while achieving a 99.20% attack success rate. This means users would likely notice no degradation in service until the specific "click" trigger is activated.

Conclusion and Defenses

The modularity of modern SpeechLLMs, while efficient for training via techniques like LoRA, creates multiple points of failure. An attacker only needs to compromise one part of the supply chain—such as a widely used pretrained encoder—to infect the entire system.

While the study found that full fine-tuning on a guaranteed clean dataset can effectively "wash out" the backdoor (reducing AER significantly), this process is computationally expensive. Partial fine-tuning often fails to remove the vulnerability. Future work is needed to develop more cost-effective detection and mitigation strategies for these subtle yet powerful multimodal threats.

Summary updated

Abstract

Large Language Models (LLMs) and their multimodal extensions are becoming increasingly popular. One common approach to enable multimodality is to cascade domain-specific encoders with an LLM, making the resulting model inherit vulnerabilities from all of its components. In this work, we present the first systematic study of audio backdoor attacks against speech language models. We demonstrate its effectiveness across four speech encoders and three datasets, covering four tasks: automatic speech recognition (ASR), speech emotion recognition, and gender and age prediction. The attack consistently achieves high success rates, ranging from 90.76\% to 99.41\%. To better understand how backdoors propagate, we conduct a component-wise analysis to identify the most vulnerable stages of the pipeline. Finally, we propose a fine-tuning-based defense that mitigates the threat of poisoned pretrained encoders.
BibTeX
@misc{fortier2025backdoor,
  author = {Alexandrine Fortier and Thomas Thebaud and Jesús Villalba and Najim Dehak and Patrick Cardinal},
  title = {{Backdoor Attacks Against Speech Language Models}},
  year = {2025},
  eprint = {2510.01157},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.01157},
}
Figure