TL;DR
AudioJev is a unified audio-language model that makes direct decisions from waveforms without needing a transcript. It introduces a novel training objective called Random-Derangement Symmetric KL (RD-SKL) to solve "order bias," ensuring that the model’s probability estimates remain stable even if the sequence of multiple-choice answer options is shuffled.
What Problem Does This Paper Address?
Most modern audio assistants follow a "transcribe-then-classify" pipeline. However, transcripts are lossy: they strip away crucial information like environmental sounds, musical properties, and conversational timing. While direct audio-to-intent models exist, they often suffer from order sensitivity.
If you ask a model to identify a sound and provide options (A: Dog, B: Car, C: Rain), the model might give "Dog" a high probability. But if you simply reorder the list (A: Rain, B: Dog, C: Car), the model’s internal bias toward specific positions (like always favoring option 'A') might cause it to change its mind. AudioJev aims to create a direct audio interface that is both highly accurate across diverse tasks and robust to how choices are presented.
Methods
AudioJev is built on the Qwen2.5-Omni 3B backbone. It maps a raw waveform, a natural language question, and a list of candidate answers directly to a probability distribution.
The core innovation is the Random-Derangement SKL (RD-SKL) training method:
- Paired Views: During training, the model sees two "views" of every question. The first is the original; the second is a "derangement" where every answer option is moved to a new position.
- Identity Alignment: The model aligns the output probabilities from both views back to the original answer identities.
- Consistency Penalty: It applies a symmetric Kullback-Leibler (KL) divergence penalty between these two distributions. This forces the model to learn that the meaning of the answer is what matters, not its position in the list.
Crucially, this calibration happens during training. At inference time, the model requires only a single forward pass, making it much faster than ensemble methods or complex calibration heads.
Datasets
The authors trained AudioJev on a diverse mixture of seven capability families:
- Acoustic/Events: ESC-50, TUT2018.
- Speech/Intent: SLURP, MINDS.
- Music: NSynth.
- Reasoning/QA: Clotho-AQA, MMAU (10,000 questions), and MMAR (1,000 questions).
- Conversational: SpokenWOZ and Oration (for voice activity and transition detection).
Key Insights and Ideas
- Direct Audio > Transcripts: For tasks involving non-speech sounds or music, direct waveform models significantly outperform text-based cascades because the transcript simply doesn't contain the necessary evidence.
- Order Bias is Quantifiable: The paper uses "Random SKL" and "Flip Rate" to measure how much a model's confidence and final decision change when options are shuffled. Without RD-SKL, even high-performing models are surprisingly fickle.
- Unified Interface: Instead of having separate "heads" for intent, sound detection, and speaker turn-taking, AudioJev uses a single shared parameter set. The task is defined entirely by the question and the provided alternatives.
Key Results
- Superior Accuracy: AudioJev achieved 68.88% on MMAU and 55.33% on MMAR, outperforming the strongest transcript-based pipeline (Whisper + Qwen3.5-4B) by several percentage points.
- Massive Speedup: By bypassing the transcription stage, AudioJev is 19.3x faster than ASR-text cascades, averaging just 62.4ms per request.
- Robust Calibration: RD-SKL reduced probability divergence (SKL) by 43.8% to 60.5% compared to standard training, significantly lowering the rate at which shuffling options "flipped" the model's answer.
In conversational tasks, the model showed impressive zero-shot-like performance, reaching a 0.900 AUROC for predicting next-speaker changes in entirely new sessions, as shown in the table below:
Limitations and Future Work
- Closed-Set Constraint: AudioJev is designed for "typed decisions" where the possible answers are known beforehand. It is not intended for open-ended generation or transcription.
- Offline Conversational Eval: While the model performs well on conversational datasets, the authors note that real-world deployment in a full-duplex voice agent involves additional complexities like interruption costs and latency thresholds that were not the focus of this study.
- Paraphrasing: While the model is robust to reordering, the authors note that robustness to paraphrasing (e.g., "Dog" vs. "Canine") is a related but distinct challenge for future research.
Summary updated