Abstract
Speech Language Models (SLMs) enable natural interactions via spoken instructions, which more effectively capture user intent by detecting nuances in speech. The richer speech signal introduces new security risks compared to text-based models, as adversaries can better bypass safety mechanisms by injecting imperceptible noise to speech. We analyze adversarial attacks under white-box access and find that SLMs are substantially more vulnerable to jailbreak attacks, which can achieve a perfect 100% attack success rate in some instances. To improve security, we propose post-hoc patching defenses used to intervene during inference by modifying the SLM’s activations that improve robustness up to 99% with (i) negligible impact on utility and (ii) without any re-training. We conduct ablation studies to maximize the efficacy of our defenses and improve the utility/security trade-off, validated with large-scale benchmarks unique to SLMs.
Introduction
The transition from text-only Large Language Models (LLMs) to Speech Language Models (SLMs) has introduced a significant security challenge. Unlike discrete text tokens, speech is a continuous, high-dimensional signal. This characteristic allows attackers to inject adversarial noise—subtle perturbations invisible to humans—that can systematically steer a model toward generating harmful content, such as instructions for illegal activities.
Research indicates that safety alignments established for text models do not inherently transfer to the audio modality. Standard defenses, such as pre-processing audio with denoisers, often result in a severe "utility/security trade-off," where the model's ability to understand legitimate speech is significantly degraded. The SPIRIT framework addresses this by providing a method to secure SLMs without requiring costly retraining or sacrificing core functionality.
The SPIRIT Framework
SPIRIT introduces post-hoc defense mechanisms that intervene during the model's forward pass. The core of the approach lies in identifying "noise-sensitive neurons" within the model's MLP layers—those that exhibit the most extreme reactions to adversarial noise compared to clean audio.
The framework evaluates three primary strategies:
- Activation Patching: The flagship method where activations of the top-$k$ most sensitive neurons are replaced during inference with activations derived from a denoised version of the same input.
- Bias Addition: Adding a constant bias to stabilize sensitive neurons.
- Neuron Pruning: Zeroing out the specific neurons exploited by the attack.
Experimental Results and Insights
The researchers demonstrated that without defense, white-box attacks using Projected Gradient Descent (PGD) achieved a 100% success rate in eliciting harmful responses across categories like "Bomb Explosive" and "Hack Information."
A key discovery was the disparity between patching the Audio Encoder versus the Language Model (LM) component. The LM proved significantly more robust to internal interventions. Patching just the top 5% of sensitive neurons in the LM allowed the defense to reach a 99% Defense Success Rate (DSR) while maintaining or even slightly improving the model's original utility score (GPT-Score).
In contrast, interventions in the audio encoder were found to be "fragile." For instance, while standard denoising dropped the GPT-Score from 6.77 to 6.28, SPIRIT’s LM-Patching maintained it at 6.80-6.83. Furthermore, on the LibriSpeech benchmark, SPIRIT maintained a Word Error Rate (WER) of ~3.15%, whereas noise-addition defenses caused the WER to skyrocket to over 27%, rendering the model ineffective for transcription.
Conclusion
SPIRIT demonstrates that targeted, post-hoc interventions can effectively neutralize jailbreak attacks in SLMs. By surgically patching a small fraction of neurons in the language model, it is possible to achieve near-perfect safety without degrading performance. While current implementation requires additional inference time for generating reference activations, future work will explore hybrid strategies and robustness against transfer attacks to further harden Speech Language Models for real-world deployment.
Summary updated