AMUSE: Audio-Visual Benchmark and Alignment Framework for Agentic Multi-Speaker Understanding

Sanjoy Chowdhury, Karren D. Yang, Xudong Liu, Fartash Faghri, Pavan Kumar Anasosalu Vasu, Oncel Tuzel, Dinesh Manocha, Chun-Liang Li, Raviteja VemulapalliarXiv 2025 · DOI · arXiv

Introduction

Multimodal Large Language Models (MLLMs) have demonstrated impressive capabilities in image description and basic video instruction following. However, complex social environments—such as crowded rooms with overlapping speakers—remain a significant challenge. Models often struggle to track speaker identities, ground dialogue temporally, and maintain coherence across scenes. To address these limitations, researchers from the University of Maryland and Apple introduce AMUSE, a benchmark for "agentic" multi-speaker skills, and RAFT, a training framework designed to emulate human-like reasoning in social interactions.

The Perception-Reasoning Gap

Current MLLMs often exhibit a "perception-reasoning gap." While they can perceive faces and voices, they frequently fail at agentic reasoning: the capacity to decompose high-level goals (e.g., "summarize the red-shirted speaker's points") into actionable sub-steps like planning, grounding, and reflection. Existing benchmarks typically focus on single-speaker or simple QA scenarios, ignoring the complexities of real-world human conversation such as rapid turn-taking and overlapping speech.

Methods: Evaluation and Alignment

The researchers propose a dual approach involving a new evaluation paradigm and a specialized alignment framework.

The AMUSE Evaluation Modes

AMUSE assesses models across three levels of autonomy:

  1. Zero-Shot: Raw video/audio input with a direct question.
  2. Guided: Provided with "scaffolding" such as transcripts and face crops.
  3. Agentic: The most challenging mode, where the model must autonomously utilize a toolset (e.g., Whisper for ASR, InsightFace for face tracking) to synthesize an answer.

The RAFT Framework

To bridge the performance gap, the RAFT (Reasoning-Acting-Feedback Training) framework introduces three key innovations:

  • Structured Reasoning Alignment: Trains models to follow a "Plan-Act-Reflect" loop.
  • Reflective Reward Optimization (RRO): An intrinsic reward system where the model evaluates its own internal consistency.
  • Selective Reasoning Adaptation (SRA): A data-efficient method that updates only cross-modal reasoning layers (approx. 0.5% of parameters), preserving general knowledge while enhancing specialized reasoning.

The AMUSE Benchmark

The AMUSE dataset consists of 2,100 samples across six task families, curated from diverse sources like AVA Active Speaker, VoxCeleb2, and FriendsMMC.

Overview of the six task families in the AMUSE benchmark: Audio-Visual Dialogue Summarization, Speaker Association, Next Speaker Prediction, Speaker Re-identification, Speaker Temporal Grounding, and Cross-scene Narrative Linking.

The tasks include:

  1. Audio-Visual Dialogue Summarization (AVDS): Attributing content to specific speakers.
  2. Audio-Visual Speaker Association (AVSA): Matching voices to visible faces.
  3. Next Speaker Prediction (NSP): Anticipating turn-taking based on social cues.
  4. Speaker Re-identification (SRID): Recognizing individuals across different clips.
  5. Speaker Temporal Grounding (STG): Precisely timing speaker turns.
  6. Cross-Scene Narrative Linking (CSNL): Connecting events across disjointed video segments.

Key Results and Insights

  • The Autonomy Penalty: Models perform significantly worse in "Agentic" mode compared to "Guided" mode, highlighting a reliance on external scaffolding rather than internal reasoning.
  • RAFT Performance Gains: RAFT achieved up to a 39.52% relative improvement on the AMUSE benchmark. Notably, it allowed open-source models like Qwen3-Omni to rival or exceed the performance of closed-source models like GPT-4o.
  • Efficiency of SRA: Updating only 0.5% of parameters via SRA outperformed standard LoRA (5% updates), suggesting that targeted cross-modal alignment is more effective than broad fine-tuning.
  • Multimodal Necessity: Ablation studies confirmed that both audio and video are essential; removing either modality leads to substantial performance degradation in multi-speaker contexts.

Conclusion and Future Work

While RAFT significantly improves multi-speaker understanding, extreme scenarios with three or more overlapping voices remain a frontier challenge. Future research may explore scaling agentic behavior with larger tool libraries and applying these principles to embodied AI for real-time human-robot interaction.

Summary updated

Abstract

Recent multimodal large language models (MLLMs) such as GPT-4o and Qwen3-Omni show strong perception but struggle in multi-speaker, dialogue-centric settings that demand agentic reasoning tracking who speaks, maintaining roles, and grounding events across time. These scenarios are central to multimodal audio-video understanding, where models must jointly reason over audio and visual streams in applications such as conversational video assistants and meeting analytics. We introduce AMUSE, a benchmark designed around tasks that are inherently agentic, requiring models to decompose complex audio-visual interactions into planning, grounding, and reflection steps. It evaluates MLLMs across three modes zero-shot, guided, and agentic and six task families, including spatio-temporal speaker grounding and multimodal dialogue summarization. Across all modes, current models exhibit weak multi-speaker reasoning and inconsistent behavior under both non-agentic and agentic evaluation. Motivated by the inherently agentic nature of these tasks and recent advances in LLM agents, we propose RAFT, a data-efficient agentic alignment framework that integrates reward optimization with intrinsic multimodal self-evaluation as reward and selective parameter adaptation for data and parameter efficient updates. Using RAFT, we achieve up to 39.52\% relative improvement in accuracy on our benchmark. Together, AMUSE and RAFT provide a practical platform for examining agentic reasoning in multimodal models and improving their capabilities.
BibTeX
@misc{chowdhury2025amuse,
  author = {Sanjoy Chowdhury and Karren D. Yang and Xudong Liu and Fartash Faghri and Pavan Kumar Anasosalu Vasu and Oncel Tuzel and Dinesh Manocha and Chun-Liang Li and Raviteja Vemulapalli},
  title = {{AMUSE: Audio-Visual Benchmark and Alignment Framework for Agentic Multi-Speaker Understanding}},
  year = {2025},
  eprint = {2512.16250},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.16250},
}
Figure