TL;DR
The paper introduces Audio-Text Interleaved Contextual Retrieval (ATIR), a new task and benchmark that moves beyond single-turn interactions to model complex, multi-turn dialogues where audio and text modalities alternate. To solve this, the authors developed ATIR-Qwen-3B, a model featuring a novel "token selector" that intelligently filters redundant audio information, achieving superior retrieval accuracy and significantly lower latency than traditional ASR-based pipelines.
What Problem Does This Paper Address?
Most multimodal retrieval research focuses on static, single-turn interactions—think of searching for a song based on a short audio clip or a text description. However, real-world communication is dynamic and interleaved. Consider a conversational assistant where a user switches between typing and speaking, or a lecture recording that combines spoken words with textual slides.
Existing retrievers struggle here for two main reasons:
- Contextual Dependency: They aren't designed to capture the semantic flow across alternating modalities in a multi-turn conversation.
- Audio Redundancy: Audio data is far less dense than text. A few seconds of audio generates hundreds of tokens, many of which are silence or noise. Feeding all these into a Multimodal Large Language Model (MLLM) is computationally expensive and introduces noise that degrades retrieval performance.
Methods
The authors propose a bi-encoder architecture based on the Qwen2.5-Omni-3B backbone. The core innovation is the ATIR Selector, a plug-and-play module designed to handle audio redundancy.
- The Selector: Instead of using all audio tokens or simple uniform pooling (which is sensitive to noise), the authors added a lightweight linear layer on top of a small LLM (Qwen3-0.6B). This selector predicts the importance of each audio token. Tokens with a probability above a certain threshold are kept, while redundant or noisy ones are discarded.
- Supervised Training: The selector is trained using timestamp-level annotations from the SVQ dataset, learning to identify "salient" segments of audio.
- Two-Stage Training Pipeline:
- Embedding Ability Activation: Training on simple text-text and audio-audio pairs to stabilize cross-modal alignment.
- Interleaved-modal Capability Elicitation: Training on complex, multi-turn interleaved data with "hard negatives" (plausible but incorrect documents) to sharpen the model's discriminative power.
Datasets
Because no benchmark existed for this specific task, the authors created the ATIR Benchmark using a rigorous synthesis pipeline:
- Source Data: They integrated three diverse datasets: LibriSpeech (ASR), CoQA (Conversational QA), and SVQ (Audio Retrieval).
- Synthesis: They used MLLMs to expand source content into semantically rich documents and generate multi-turn, interleaved queries grounded in those documents.
- Robustness: To simulate real-world conditions, they synthesized audio across four environments: clean, speech noise, traffic noise, and media noise.
- Scale: The final dataset contains over 88,000 annotated query-positive document pairs.
Key Insights and Ideas
- Selective Filtering > Uniform Pooling: The ATIR Selector significantly outperforms standard average pooling. By focusing only on informative tokens, the model avoids being misled by background noise or silence.
- Sequential Alignment is Crucial: The authors performed "perturbation studies" where they shuffled the order of audio and text turns. Performance dropped sharply, proving that the model is truly learning the contextual relationship between turns, not just looking for keywords across modalities.
- Direct Audio Processing Wins: Traditional pipelines that transcribe audio to text (ASR) before retrieval are not only slower but also lose non-textual information like speaker traits and emotion. ATIR-Qwen-3B processes audio directly, preserving these cues.
Key Results
- State-of-the-Art Performance: ATIR-Qwen-3B consistently outperformed text-modal, cross-modal, and fused-modal baselines. It beat the strongest text-modal baseline (which used high-quality Whisper transcriptions) by +9.52% in Recall@1.
- Efficiency: Because it avoids the overhead of a separate ASR transcription step, ATIR-Qwen-3B achieved an average latency of just 16.8ms per query, compared to over 500ms for ASR-based text retrievers.
- Ablation Success: Removing the ATIR Selector led to a consistent performance drop, validating the benefit of selective audio token filtering.
Limitations and Future Work
The authors acknowledge several areas for growth:
- Model Expressiveness: The current framework uses a relatively lightweight design for efficiency; more expressive (and likely larger) models could push performance further.
- Multi-Document Retrieval: The current task focuses on retrieving a single relevant document. Future work could explore complex RAG (Retrieval-Augmented Generation) scenarios requiring evidence from multiple sources.
- Task Breadth: While the benchmark is QA-centric, the authors hope to extend the framework to a broader range of multimodal tasks and real-world application scenarios.
Summary updated