Abstract
Audiovisual video captioning aims to generate semantically rich descriptions with temporal alignment between visual and auditory events, thereby benefiting both video understanding and generation. In this paper, we present AVoCaDO, a powerful audiovisual video captioner driven by the temporal orchestration between audio and visual modalities. We propose a two-stage post-training pipeline: (1) AVoCaDO SFT, which fine-tunes the model on a newly curated dataset of 107K high-quality, temporally-aligned audiovisual captions; and (2) AVoCaDO GRPO, which leverages tailored reward functions to further enhance temporal coherence and dialogue accuracy while regularizing caption length and reducing collapse. Experimental results demonstrate that AVoCaDO significantly outperforms existing open-source models across four audiovisual video captioning benchmarks, and also achieves competitive performance on the VDC and DREAM-1K benchmark under visual-only settings.
Introduction
Most contemporary video captioning models are "vision-centric," excelling at describing on-screen actions but often neglecting the rich semantic cues found in audio, such as dialogue, background music, and environmental sound effects. Existing attempts to integrate audio often rely on a "decoupled" strategy—concatenating separate visual and auditory descriptions—which fails to capture the fine-grained temporal alignment and causal interplay between sight and sound. AVoCaDO addresses this by integrating and reasoning over both modalities simultaneously through a novel temporal orchestration framework.
Methodology
AVoCaDO is built upon the Qwen2.5-Omni-7B architecture and refined through a two-stage post-training pipeline designed to maximize audiovisual synergy.
Data Curation and Supervised Fine-Tuning (SFT)
The researchers curated a high-quality dataset of 107K audiovisual captions. To overcome the limitations of direct joint generation, they employed a "divide and conquer" strategy: generating detailed, separate captions for video and audio streams and then using a teacher model (Gemini-2.5-Pro) to fuse them into a single, temporally synchronized narrative.
Reinforcement Learning via GRPO
To further refine the model's outputs, the authors utilized Group Relative Policy Optimization (GRPO) with three tailored reward functions:
- Checklist-based Reward ($R_C$): Uses GPT-4o to ensure the model covers key narrative and auditory points.
- Dialogue-based Reward ($R_D$): Specifically targets the accuracy of speech transcription and speaker identification.
- Length-regularized Reward ($R_L$): Penalizes repetition and encourages concise yet complete descriptions.
Experimental Results
AVoCaDO sets a new standard for open-source audiovisual captioning, outperforming existing models and rivaling proprietary systems.
The Joint Reasoning Advantage
A pilot study highlighted the necessity of joint modality processing. As shown in the performance comparison on the Daily-Omni benchmark, joint reasoning leads to a significant average accuracy gain of 15.8%, with a massive 27.8% improvement specifically in audiovisual event alignment tasks.
Benchmark Performance
- State-of-the-Art: AVoCaDO outperformed all open-source models on the video-SALMONN-2 and UGC-VideoCap benchmarks.
- Rivaling Commercial Models: On UGC-VideoCap, AVoCaDO achieved a score of 73.2, surpassing Gemini-2.5-Pro (72.6).
- Superior QA: On the Daily-Omni benchmark, AVoCaDO reached 50.1% accuracy, nearly doubling the performance of the next best open-source model.
- Visual Robustness: The model remains highly competitive on vision-only benchmarks like DREAM-1K, proving that audiovisual training enhances rather than compromises visual understanding.
Conclusion
AVoCaDO demonstrates that temporal orchestration between sight and sound is essential for comprehensive video understanding. By combining a sophisticated data curation pipeline with targeted reinforcement learning, AVoCaDO bridges the performance gap between open-source and commercial video captioning models, providing a powerful tool for the research community.
Summary updated