Rebellion: Noise-Robust Reasoning Training for Audio Reasoning Models

Tiansheng Huang, Virat Shejwalkar, Oscar Chang, Milad Nasr, Ling LiuarXiv 2025 · DOI · arXiv

Introduction

As Large Language Models (LLMs) evolve, the paradigm of "thinking before speaking"—known as Chain of Thought (CoT) reasoning—is being extended to Audio Reasoning Models (ARMs). These models process complex audio inputs and reason through them to generate responses. However, while reasoning improves general task performance, its implications for safety in the audio domain remain largely unexplored. This paper identifies a critical vulnerability in ARMs: audio jailbreak attacks that leverage "representation drift" to bypass safety guardrails. To address this, we introduce Rebellion, a noise-robust reasoning training method designed to maintain safety even under adversarial conditions.

The Challenge of Representation Drift

Standard Reasoning Training (RT) involves fine-tuning models on a mixture of benign and safety-related reasoning data. While this protects models against simple harmful queries, it fails against advanced audio jailbreaks like Advwave. These attacks inject optimized adversarial noise into the audio signal, causing representation drift. This drift shifts the model's internal activations away from the "harmful" cluster and toward a "complying" cluster, effectively blinding the safety guardrails and forcing the model to generate harmful content.

Rebellion: Robust Reasoning Training

To counter representation drift, Rebellion employs a min-max optimization strategy, conceptually similar to adversarial training. The training process involves two key steps:

  1. Inner Problem (Maximizing Loss): The model simulates the worst-case representation drift—a noise vector that maximizes safety loss—using a one-step gradient approximation.
  2. Outer Problem (Minimizing Loss): The model's weights are updated to minimize the combined safety and benign loss while under the influence of this simulated worst-case drift.

By training the model to recover from the most disruptive internal perturbations, Rebellion ensures that safety reasoning remains robust regardless of adversarial audio noise.

The "Think Twice" Phenomenon

One of the most compelling findings is the emergent "think twice" behavior in Rebellion-trained models. Attacks like Advwave often force a model to begin its response with an affirmative prefix like "Sure." In standard models, this prefix acts as a trap, compelling the model to complete the harmful request.

Figure 1: Comparison of Standard RT and Rebellion under attack. Rebellion demonstrates the "think twice" phenomenon, where the model catches its own forced mistake during the reasoning phase.

As shown in Figure 1, a Rebellion-trained model may still output the forced "Sure," but its robust safety reasoning then kicks in. The model generates a CoT that recognizes the policy violation and ultimately issues a refusal. It effectively "catches" its own forced error before delivering the final answer.

Experimental Results

Experiments conducted on the Qwen2-Audio-7B model demonstrate that Rebellion significantly enhances safety without compromising utility. Against the Advwave jailbreak, standard RT yielded a Harmful Score (HS) of 26.25%, whereas Rebellion reduced this to just 1.25%. Furthermore, Rebellion completely neutralized rephrasing attacks, dropping the HS from over 50% to 0%.

Crucially, these safety gains do not come at the cost of general intelligence. As seen in Table 1, benign accuracy on benchmarks like GSM8K and MMLU-biology remains stable or even improves under moderate Rebellion training intensities. However, there is a sensitivity threshold; if the simulated noise intensity is set too high, the optimization becomes overly difficult, leading to a decline in benign performance.

Conclusion

Rebellion represents a significant step forward in securing Audio Reasoning Models. By addressing the root cause of audio jailbreaks—representation drift—it enables models to maintain their safety integrity even when forced into initial compliance. Future work will explore extending this robust training framework from Supervised Fine-Tuning to Reinforcement Learning-based reasoning paradigms.

Summary updated

Abstract

Instilling reasoning capabilities in large models (LMs) using reasoning training (RT) significantly improves LMs' performances. Thus Audio Reasoning Models (ARMs), i.e., audio LMs that can reason, are becoming increasingly popular. However, no work has studied the safety of ARMs against jailbreak attacks that aim to elicit harmful responses from target models. To this end, first, we show that standard RT with appropriate safety reasoning data can protect ARMs from vanilla audio jailbreaks, but cannot protect them against our proposed simple yet effective jailbreaks. We show that this is because of the significant representation drift between vanilla and advanced jailbreaks which forces the target ARMs to emit harmful responses. Based on this observation, we propose Rebellion, a robust RT that trains ARMs to be robust to the worst-case representation drift. All our results are on Qwen2-Audio; they demonstrate that Rebellion: 1) can protect against advanced audio jailbreaks without compromising performance on benign tasks, and 2) significantly improves accuracy-safety trade-off over standard RT method.
BibTeX
@misc{huang2025rebellion,
  author = {Tiansheng Huang and Virat Shejwalkar and Oscar Chang and Milad Nasr and Ling Liu},
  title = {{Rebellion: Noise-Robust Reasoning Training for Audio Reasoning Models}},
  year = {2025},
  eprint = {2511.09682},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.09682},
}
Figure