AudioJev: Direct Audio Decisions with Order-Calibrated Probabilities

Sihan Lv, Zhen Li, Zhiqi Cao, Jinshan Zhang, Ying Li, Meng Xi, Jianwei YinarXiv 2026 · arXiv

TL;DR

AudioJev is a unified audio-language model that makes direct decisions from waveforms without needing a transcript. It introduces a novel training objective called Random-Derangement Symmetric KL (RD-SKL) to solve "order bias," ensuring that the model’s probability estimates remain stable even if the sequence of multiple-choice answer options is shuffled.

What Problem Does This Paper Address?

Most modern audio assistants follow a "transcribe-then-classify" pipeline. However, transcripts are lossy: they strip away crucial information like environmental sounds, musical properties, and conversational timing. While direct audio-to-intent models exist, they often suffer from order sensitivity.

If you ask a model to identify a sound and provide options (A: Dog, B: Car, C: Rain), the model might give "Dog" a high probability. But if you simply reorder the list (A: Rain, B: Dog, C: Car), the model’s internal bias toward specific positions (like always favoring option 'A') might cause it to change its mind. AudioJev aims to create a direct audio interface that is both highly accurate across diverse tasks and robust to how choices are presented.

Methods

AudioJev is built on the Qwen2.5-Omni 3B backbone. It maps a raw waveform, a natural language question, and a list of candidate answers directly to a probability distribution.

The core innovation is the Random-Derangement SKL (RD-SKL) training method:

  1. Paired Views: During training, the model sees two "views" of every question. The first is the original; the second is a "derangement" where every answer option is moved to a new position.
  2. Identity Alignment: The model aligns the output probabilities from both views back to the original answer identities.
  3. Consistency Penalty: It applies a symmetric Kullback-Leibler (KL) divergence penalty between these two distributions. This forces the model to learn that the meaning of the answer is what matters, not its position in the list.

Crucially, this calibration happens during training. At inference time, the model requires only a single forward pass, making it much faster than ensemble methods or complex calibration heads.

Figure 1: AudioJev overview. A waveform, question and caller-supplied alternatives define a decision for one shared audio classifier. Training (top): the original and randomly deranged answer lists receive candidate-only supervision, with identity-aligned symmetric KL encouraging consistent probabilities across presentations. Inference (bottom): one forward pass scores the supplied alternatives and returns their probability distribution, supporting different audio tasks through the same interface.

Datasets

The authors trained AudioJev on a diverse mixture of seven capability families:

  • Acoustic/Events: ESC-50, TUT2018.
  • Speech/Intent: SLURP, MINDS.
  • Music: NSynth.
  • Reasoning/QA: Clotho-AQA, MMAU (10,000 questions), and MMAR (1,000 questions).
  • Conversational: SpokenWOZ and Oration (for voice activity and transition detection).

Key Insights and Ideas

  • Direct Audio > Transcripts: For tasks involving non-speech sounds or music, direct waveform models significantly outperform text-based cascades because the transcript simply doesn't contain the necessary evidence.
  • Order Bias is Quantifiable: The paper uses "Random SKL" and "Flip Rate" to measure how much a model's confidence and final decision change when options are shuffled. Without RD-SKL, even high-performing models are surprisingly fickle.
  • Unified Interface: Instead of having separate "heads" for intent, sound detection, and speaker turn-taking, AudioJev uses a single shared parameter set. The task is defined entirely by the question and the provided alternatives.

Key Results

  • Superior Accuracy: AudioJev achieved 68.88% on MMAU and 55.33% on MMAR, outperforming the strongest transcript-based pipeline (Whisper + Qwen3.5-4B) by several percentage points.
  • Massive Speedup: By bypassing the transcription stage, AudioJev is 19.3x faster than ASR-text cascades, averaging just 62.4ms per request.
  • Robust Calibration: RD-SKL reduced probability divergence (SKL) by 43.8% to 60.5% compared to standard training, significantly lowering the rate at which shuffling options "flipped" the model's answer.
Comparison of AudioJev against direct audio models and Whisper-based cascades.

In conversational tasks, the model showed impressive zero-shot-like performance, reaching a 0.900 AUROC for predicting next-speaker changes in entirely new sessions, as shown in the table below:

Causal next-speaker prediction results.

Limitations and Future Work

  • Closed-Set Constraint: AudioJev is designed for "typed decisions" where the possible answers are known beforehand. It is not intended for open-ended generation or transcription.
  • Offline Conversational Eval: While the model performs well on conversational datasets, the authors note that real-world deployment in a full-duplex voice agent involves additional complexities like interruption costs and latency thresholds that were not the focus of this study.
  • Paraphrasing: While the model is robust to reordering, the authors note that robustness to paraphrasing (e.g., "Dog" vs. "Canine") is a related but distinct challenge for future research.

Summary updated

Abstract

Audio decisions often depend on evidence that a transcript does not preserve, while their probability estimates can depend on how answer options are ordered. AudioJev maps a waveform, question and supplied alternatives directly to a candidate distribution through one shared full-parameter model. We define order calibration as preserving an answer's probability under meaning-preserving option permutations. Random-derangement SKL training pairs each question with a reordered view in which every alternative changes position, supervises both answers, and aligns the two distributions before applying a symmetric KL penalty. Inference retains a single candidate-scoring forward, with no calibration head or order ensemble. Across three training seeds, AudioJev reaches 68.88%/55.33% mean accuracy on complete MMAU/MMAR and reduces random-order SKL by 43.8%/60.5% relative to the single-view removal ablation. The same model handles intent, environmental sound, note properties, speech activity and conversational transitions. Paired removal ablations and multi-order evaluation measure predictive accuracy and probability stability together, establishing a direct audio interface whose calibration objective acts on candidate meaning rather than presentation position. Inference code and model weights are available at https://github.com/SihanLv/AudioJev-Inference and https://huggingface.co/shlv/AudioJev.
BibTeX
@misc{lv2026audiojev,
  author = {Sihan Lv and Zhen Li and Zhiqi Cao and Jinshan Zhang and Ying Li and Meng Xi and Jianwei Yin},
  title = {{AudioJev: Direct Audio Decisions with Order-Calibrated Probabilities}},
  year = {2026},
  eprint = {2610.01293},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.01293},
}
Figure