Vox-Infinity: Benchmarking the Limits of Long-Context Spoken Language Models

Xize Cheng, Wenxu Jia, Chenyuhao Wen, Dongjie Fu, Zehan Wang, Xinyu Zhang, Tao Jin

TL;DR

The authors introduce Vox-Infinity, the first benchmark specifically designed to evaluate long-context understanding in spoken language models (SLMs). By evaluating seven representative SLMs, the paper reveals a pronounced "recency effect"—where models struggle to retrieve information located deep in dialogue history—and highlights a stark trade-off between compact text transcripts and raw audio histories.

What Problem Does This Paper Address?

While Large Language Models (LLMs) have rapidly expanded their context windows to millions of tokens, Spoken Language Models (SLMs) face a unique architectural bottleneck: audio is a low-compression modality. Encoding just one second of speech typically requires 12 to 25 embeddings. Consequently, an audio dialogue accumulates sequence length much faster than its textual equivalent, severely taxing the model's long-context attention mechanisms.

Prior audio benchmarks mostly focused on pushing past the traditional 30-second limit of standard audio encoders (like Whisper). However, they lacked a systematic way to evaluate how well a model can reason over minutes or hours of interleaved dialogue, or how well it retains non-verbal, paralinguistic cues (like emotion and intonation) over time.

Methods

The authors constructed Vox-Infinity to systematically scale audio history along two axes: turn count and turn duration.

A key methodological innovation is the introduction of explicit answer-provenance annotations. Instead of just measuring total session duration, the benchmark tracks the exact timestamp of the earliest evidence required to answer a query. This allows the evaluation to be grouped into precise "required context duration" bins (e.g., 0–2 min, 2–4 min, 4–8 min, 8m+).

The evaluation framework tests models under two distinct paradigms:

  1. Text-as-history: The dialogue history is provided as text transcripts (generated via an ASR model).
  2. Audio-as-history: The dialogue history is provided entirely as raw audio sequences.

To evaluate model outputs, the framework leverages GPT-4o as an LLM judge, categorizing answers into Correct, Incorrect, or Unknown, which was validated to have a 99% agreement rate with human annotators.

Datasets

The paper introduces the Vox-Infinity benchmark, which aggregates 1,474 sessions, 13,516 QA pairs, and 127.96 hours of audio. It is split into three distinct scenarios designed to test different aspects of long-context speech processing:

Illustration of the three dialogue scenarios in Vox-Infinity.
  • Ultra Multi-Turn Dialogue: Sourced from SpokenWOZ, featuring dense, back-and-forth task-oriented conversations (more than 20 turns) requiring fine-grained information retrieval (e.g., phone numbers, train IDs).
  • Personal Monologue: Long-form single-speaker speeches (longer than 2 minutes) reconstructed from GigaSpeech, requiring multi-hop reasoning over distributed evidence.
  • Beyond-Semantic Dialogue: Built using IEMOCAP and augmented with Qwen-TTS synthesized speech. It features semantically neutral text spoken with specific emotional styles to isolate acoustic emotion recognition from text-based semantic inference.

Key Insights and Ideas

  • The Recency Effect is Real: Across all models, accuracy drops noticeably as the supporting evidence moves further back in the timeline. Models are simply better at remembering what happened recently.
  • The Modality Trade-Off: For purely semantic tasks, providing history as text transcripts drastically outperforms raw audio. Text is compact and preserves exact lexical tokens, whereas long audio streams overwhelm the models' retrieval capabilities.
  • Audio is Essential for Tone: When it comes to tracking paralinguistic information (like a speaker's emotional state over a long conversation), text transcripts fail because they strip away prosody, rhythm, and intensity. In these scenarios, raw audio history proves superior, particularly when the literal text is emotionally neutral.
Fine-grained accuracy under audio history showing the recency effect across different time intervals.

Key Results

The authors evaluated seven prominent models, including Qwen3-Omni, GLM-4-Voice, AudioFlamingo3, Gemini-3.1-Pro, and Qwen3.5-Plus.

  • Semantic Retrieval Gap: On the Ultra Multi-Turn subset, the average accuracy using text history was 71.3%, compared to a dismal 42.6% when using audio history. This shows that current SLMs struggle to pinpoint precise details (like numbers or names) within long audio files.
  • Paralinguistic Superiority: On the Beyond-Semantic controlled (synthesized) samples, audio history outperformed text history for six out of seven models, raising the model-averaged accuracy from 23.8% (text) to 30.4% (audio).
  • The GLM-4-Voice Exception: Interestingly, GLM-4-Voice performed better with text history even on emotional tasks. The authors attribute this to its discrete speech tokenizer, which prioritizes linguistic content over emotional prosody.

Limitations and Future Work

While Vox-Infinity exposes critical vulnerabilities in current SLMs, the authors note a few limitations and areas for development:

  • The Audio Compression Bottleneck: Current audio tokenizers either lose too much fine-grained semantic detail or generate sequences that are too long for LLM context windows. Future work must focus on developing high-compression, feature-rich audio codecs.
  • Synthetic Data Dependency: Part of the emotional dialogue dataset relies on TTS-synthesized speech to cleanly isolate semantic content from acoustic style. Testing on more diverse, naturally occurring emotional long-form audio remains an open challenge.

Summary updated

Abstract

Long-context understanding remains a fundamental challenge for large language models, as excessively long inputs often lead models to forget salient information. This issue is even more pronounced in the speech domain, where audio, as a low-compression modality, requires substantially more embeddings than text to preserve both semantic content and acoustic cues. To address this challenge, we introduce \textbf{Vox-Infinity}, the first benchmark specifically designed to evaluate long-context understanding in spoken language models. Vox-Infinity systematically extends audio history along two dimensions: turn count and turn duration. It covers a diverse range of representative scenarios with varying interaction structures and semantic complexity. Crucially, Vox-Infinity provides explicit answer-provenance annotations and organizes samples according to the amount of historical context required to resolve each query, enabling precise and length-aware evaluation. Extensive evaluations of seven representative spoken language models reveal a clear overall recency effect: models generally achieve higher accuracy when answer-supporting evidence is closer to the query, but struggle to retrieve and use evidence located farther back in the dialogue history. Cases and datasets are available at https://vox-infinity.github.io.
BibTeX
@misc{cheng2026vox,
  author = {Xize Cheng and Wenxu Jia and Chenyuhao Wen and Dongjie Fu and Zehan Wang and Xinyu Zhang and Tao Jin},
  title = {{Vox-Infinity: Benchmarking the Limits of Long-Context Spoken Language Models}},
  year = {2026},
  eprint = {2609.22452},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.22452},
}
Figure