ATIR: Towards Audio-Text Interleaved Contextual Retrieval

Tong Zhao, Chenghao Zhang, Yutao Zhu, Zhicheng Dou

TL;DR

The paper introduces Audio-Text Interleaved Contextual Retrieval (ATIR), a new task and benchmark that moves beyond single-turn interactions to model complex, multi-turn dialogues where audio and text modalities alternate. To solve this, the authors developed ATIR-Qwen-3B, a model featuring a novel "token selector" that intelligently filters redundant audio information, achieving superior retrieval accuracy and significantly lower latency than traditional ASR-based pipelines.

What Problem Does This Paper Address?

Most multimodal retrieval research focuses on static, single-turn interactions—think of searching for a song based on a short audio clip or a text description. However, real-world communication is dynamic and interleaved. Consider a conversational assistant where a user switches between typing and speaking, or a lecture recording that combines spoken words with textual slides.

Existing retrievers struggle here for two main reasons:

  1. Contextual Dependency: They aren't designed to capture the semantic flow across alternating modalities in a multi-turn conversation.
  2. Audio Redundancy: Audio data is far less dense than text. A few seconds of audio generates hundreds of tokens, many of which are silence or noise. Feeding all these into a Multimodal Large Language Model (MLLM) is computationally expensive and introduces noise that degrades retrieval performance.
Figure 1: Comparison of traditional cross-modal and fused-modal retrieval settings with our proposed ATIR paradigm, where queries contain alternating segments that require contextual, multi-turn understanding.

Methods

The authors propose a bi-encoder architecture based on the Qwen2.5-Omni-3B backbone. The core innovation is the ATIR Selector, a plug-and-play module designed to handle audio redundancy.

  • The Selector: Instead of using all audio tokens or simple uniform pooling (which is sensitive to noise), the authors added a lightweight linear layer on top of a small LLM (Qwen3-0.6B). This selector predicts the importance of each audio token. Tokens with a probability above a certain threshold are kept, while redundant or noisy ones are discarded.
  • Supervised Training: The selector is trained using timestamp-level annotations from the SVQ dataset, learning to identify "salient" segments of audio.
  • Two-Stage Training Pipeline:
    1. Embedding Ability Activation: Training on simple text-text and audio-audio pairs to stabilize cross-modal alignment.
  • Interleaved-modal Capability Elicitation: Training on complex, multi-turn interleaved data with "hard negatives" (plausible but incorrect documents) to sharpen the model's discriminative power.
Figure 3: Architecture of the ATIR-Qwen-3B and the training paradigm of the ATIR Selector. The selector is plugged into the backbone to identify informative audio tokens and filter redundant context. It is trained with supervision derived from timestamp-level annotations, enabling selective context modeling and efficient audio–text interleaved representation learning.

Datasets

Because no benchmark existed for this specific task, the authors created the ATIR Benchmark using a rigorous synthesis pipeline:

  • Source Data: They integrated three diverse datasets: LibriSpeech (ASR), CoQA (Conversational QA), and SVQ (Audio Retrieval).
  • Synthesis: They used MLLMs to expand source content into semantically rich documents and generate multi-turn, interleaved queries grounded in those documents.
  • Robustness: To simulate real-world conditions, they synthesized audio across four environments: clean, speech noise, traffic noise, and media noise.
  • Scale: The final dataset contains over 88,000 annotated query-positive document pairs.
Figure 2: Overview of the ATIR dataset construction pipeline. The pipeline comprises data configuration and multi- grained corpus generation, query–answer pair construction with reasoning-oriented queries and source-grounded answers, hard negative mining for retrieval, and iterative self-evaluation for quality refinement.

Key Insights and Ideas

  • Selective Filtering > Uniform Pooling: The ATIR Selector significantly outperforms standard average pooling. By focusing only on informative tokens, the model avoids being misled by background noise or silence.
  • Sequential Alignment is Crucial: The authors performed "perturbation studies" where they shuffled the order of audio and text turns. Performance dropped sharply, proving that the model is truly learning the contextual relationship between turns, not just looking for keywords across modalities.
  • Direct Audio Processing Wins: Traditional pipelines that transcribe audio to text (ASR) before retrieval are not only slower but also lose non-textual information like speaker traits and emotion. ATIR-Qwen-3B processes audio directly, preserving these cues.

Key Results

  • State-of-the-Art Performance: ATIR-Qwen-3B consistently outperformed text-modal, cross-modal, and fused-modal baselines. It beat the strongest text-modal baseline (which used high-quality Whisper transcriptions) by +9.52% in Recall@1.
  • Efficiency: Because it avoids the overhead of a separate ASR transcription step, ATIR-Qwen-3B achieved an average latency of just 16.8ms per query, compared to over 500ms for ASR-based text retrievers.
  • Ablation Success: Removing the ATIR Selector led to a consistent performance drop, validating the benefit of selective audio token filtering.

Limitations and Future Work

The authors acknowledge several areas for growth:

  • Model Expressiveness: The current framework uses a relatively lightweight design for efficiency; more expressive (and likely larger) models could push performance further.
  • Multi-Document Retrieval: The current task focuses on retrieving a single relevant document. Future work could explore complex RAG (Retrieval-Augmented Generation) scenarios requiring evidence from multiple sources.
  • Task Breadth: While the benchmark is QA-centric, the authors hope to extend the framework to a broader range of multimodal tasks and real-world application scenarios.

Summary updated

Abstract

Audio carries richer information than text, including emotion, speaker traits, and environmental context, while also enabling lower-latency processing compared to speech-to-text pipelines. However, recent multimodal information retrieval research has predominantly focused on images, largely overlooking audio, especially in the setting of interleaved audio-text contextual retrieval. In this work, we introduce the Audio-Text Interleaved contextual Retrieval (ATIR) task, where queries can alternate between audio and text modalities. We construct an ATIR benchmark by integrating several Automatic Speech Recognition (ASR), QA, and retrieval datasets, ultimately unifying four types of contextual retrieval tasks. This benchmark substantially addresses the limitations of existing audio retrieval datasets in semantic retrieval. To study this task, we evaluate several off-the-shelf retrievers and train our ATIR model based on a Multimodal Large Language Model (MLLM). We further introduce a novel token compression mechanism that is orthogonal to existing compression methods, thereby alleviating the issue of excessive audio tokens in MLLM-based ATIR models. Experimental results demonstrate that our ATIR model achieves substantial improvements over strong baselines.
BibTeX
@inproceedings{zhao2026atir,
  author = {Tong Zhao and Chenghao Zhang and Yutao Zhu and Zhicheng Dou},
  title = {{ATIR: Towards Audio-Text Interleaved Contextual Retrieval}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1006},
}

Loading related content...

Figure