AVoCaDO: An Audiovisual Video Captioner Driven by Temporal Orchestration

Xinlong Chen, Yue Ding, Weihong Lin, Jingyun Hua, Linli Yao, Yang Shi, Bozhou Li, Yuanxing Zhang, Qiang Liu, Pengfei Wan, Liang Wang, Tieniu Tan

Abstract

Audiovisual video captioning aims to generate semantically rich descriptions with temporal alignment between visual and auditory events, thereby benefiting both video understanding and generation. In this paper, we present AVoCaDO, a powerful audiovisual video captioner driven by the temporal orchestration between audio and visual modalities. We propose a two-stage post-training pipeline: (1) AVoCaDO SFT, which fine-tunes the model on a newly curated dataset of 107K high-quality, temporally-aligned audiovisual captions; and (2) AVoCaDO GRPO, which leverages tailored reward functions to further enhance temporal coherence and dialogue accuracy while regularizing caption length and reducing collapse. Experimental results demonstrate that AVoCaDO significantly outperforms existing open-source models across four audiovisual video captioning benchmarks, and also achieves competitive performance on the VDC and DREAM-1K benchmark under visual-only settings.

Introduction

Most contemporary video captioning models are "vision-centric," excelling at describing on-screen actions but often neglecting the rich semantic cues found in audio, such as dialogue, background music, and environmental sound effects. Existing attempts to integrate audio often rely on a "decoupled" strategy—concatenating separate visual and auditory descriptions—which fails to capture the fine-grained temporal alignment and causal interplay between sight and sound. AVoCaDO addresses this by integrating and reasoning over both modalities simultaneously through a novel temporal orchestration framework.

Methodology

AVoCaDO is built upon the Qwen2.5-Omni-7B architecture and refined through a two-stage post-training pipeline designed to maximize audiovisual synergy.

Data Curation and Supervised Fine-Tuning (SFT)

The researchers curated a high-quality dataset of 107K audiovisual captions. To overcome the limitations of direct joint generation, they employed a "divide and conquer" strategy: generating detailed, separate captions for video and audio streams and then using a teacher model (Gemini-2.5-Pro) to fuse them into a single, temporally synchronized narrative.

The AVoCaDO data curation pipeline uses a divide-and-conquer approach to fuse separate video and audio captions into a high-quality joint narrative, overseen by a quality checker.

Reinforcement Learning via GRPO

To further refine the model's outputs, the authors utilized Group Relative Policy Optimization (GRPO) with three tailored reward functions:

  • Checklist-based Reward ($R_C$): Uses GPT-4o to ensure the model covers key narrative and auditory points.
  • Dialogue-based Reward ($R_D$): Specifically targets the accuracy of speech transcription and speaker identification.
  • Length-regularized Reward ($R_L$): Penalizes repetition and encourages concise yet complete descriptions.

Experimental Results

AVoCaDO sets a new standard for open-source audiovisual captioning, outperforming existing models and rivaling proprietary systems.

The Joint Reasoning Advantage

A pilot study highlighted the necessity of joint modality processing. As shown in the performance comparison on the Daily-Omni benchmark, joint reasoning leads to a significant average accuracy gain of 15.8%, with a massive 27.8% improvement specifically in audiovisual event alignment tasks.

Comparison of separate vs. joint AV captioning. Joint reasoning significantly improves performance on alignment and reasoning tasks, as shown in the Daily-Omni benchmark results.

Benchmark Performance

  • State-of-the-Art: AVoCaDO outperformed all open-source models on the video-SALMONN-2 and UGC-VideoCap benchmarks.
  • Rivaling Commercial Models: On UGC-VideoCap, AVoCaDO achieved a score of 73.2, surpassing Gemini-2.5-Pro (72.6).
  • Superior QA: On the Daily-Omni benchmark, AVoCaDO reached 50.1% accuracy, nearly doubling the performance of the next best open-source model.
  • Visual Robustness: The model remains highly competitive on vision-only benchmarks like DREAM-1K, proving that audiovisual training enhances rather than compromises visual understanding.

Conclusion

AVoCaDO demonstrates that temporal orchestration between sight and sound is essential for comprehensive video understanding. By combining a sophisticated data curation pipeline with targeted reinforcement learning, AVoCaDO bridges the performance gap between open-source and commercial video captioning models, providing a powerful tool for the research community.

Summary updated

Abstract

Audiovisual video captioning aims to generate semantically rich descriptions with temporal alignment between visual and auditory events, thereby benefiting both video understanding and generation. In this paper, we present AVoCaDO, a powerful audiovisual video captioner driven by the temporal orchestration between audio and visual modalities. We propose a two-stage post-training pipeline: (1) AVoCaDO SFT, which fine-tunes the model on a newly curated dataset of 107K high-quality, temporally-aligned audiovisual captions; and (2) AVoCaDO GRPO, which leverages tailored reward functions to further enhance temporal coherence and dialogue accuracy while regularizing caption length and reducing collapse. Experimental results demonstrate that AVoCaDO significantly outperforms existing open-source models across four audiovisual video captioning benchmarks, and also achieves competitive performance on the VDC and DREAM-1K benchmark under visual-only settings.
BibTeX
@inproceedings{chen2026avocado,
  author = {Xinlong Chen and Yue Ding and Weihong Lin and Jingyun Hua and Linli Yao and Yang Shi and Bozhou Li and Yuanxing Zhang and Qiang Liu and Pengfei Wan and Liang Wang and Tieniu Tan},
  title = {{AVoCaDO: An Audiovisual Video Captioner Driven by Temporal Orchestration}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=vjEl1PuIDE},
}
Figure