Abstract
ViDove is a multimodal translation agent system designed to emulate the workflow of professional human translators by integrating visual, auditory, and textual context. By leveraging a sophisticated long-short term memory system and a multi-agent collaborative pipeline, ViDove addresses the limitations of "blind" and "deaf" machine translation systems. Experimental results demonstrate a 28% improvement in BLEU scores over state-of-the-art baselines for long-form video subtitling on the newly introduced DoveBench dataset.
Introduction
Large Language Models (LLMs) have significantly advanced machine translation, yet most existing systems process text in isolation, ignoring the rich multimodal context inherent in video. This lack of visual and auditory awareness often leads to translation errors where textual ambiguity cannot be resolved without external cues. For instance, the word "fold" in a culinary context could refer to folding a napkin or folding ingredients, a distinction that requires visual confirmation.
Furthermore, long-form video subtitling (exceeding 10 minutes) introduces specific challenges: maintaining terminology consistency, precise timestamp alignment, and handling domain-specific jargon. Existing Multimodal Machine Translation (MMT) systems often struggle with document-level consistency or the ability to process an entire video as a cohesive unit.
System Architecture
ViDove is an agentic framework designed to bridge the gap between AI and professional human performance through a modular, collaborative architecture.
The system comprises five specialized modules:
- Auditory Agent ($\mathcal{S}$): Utilizes state-of-the-art models to segment audio by speaker, extract emotional tone, and generate enriched transcriptions.
- Vision Agent ($\mathcal{L}^*$): Employs Vision-Language Models (VLMs) to analyze keyframes and provide high-level semantic descriptions of the visual scene.
- Memory System ($\mathcal{M}$): Features a dual-layer structure. Short-term memory stores video-specific translation history and multimodal cues for intra-video consistency. Long-term memory utilizes Retrieval-Augmented Generation (RAG) to incorporate domain-specific knowledge and web-sourced information.
- Translation Agent Team: A three-stage pipeline consisting of a Translator (initial draft), a Proofreader (terminology and grammar check), and an Editor (final quality check and visual alignment).
- Output Render: Produces the final synchronized SRT files and rendered video.
Evaluation and Benchmarks
The performance of ViDove was evaluated on two primary datasets:
- DoveBench: A new benchmark for long-form video subtitling, comprising 17 hours of human-annotated data from gaming domains (Counter-Strike and StarCraft 2). This benchmark emphasizes real-world challenges like fast-paced dialogue and technical jargon.
- BigVideo: A standard benchmark for sentence-level multimodal translation.
Key Results
ViDove establishes a new state-of-the-art for video translation across multiple metrics.
On the DoveBench dataset, ViDove achieved a BLEU score of 23.51, representing a significant improvement over the strongest baseline (Whisper + DelTA at 18.26). Additionally, the system showed a 15% improvement in Subtitle Edit Rate (SubER), indicating superior alignment between translated text and video timestamps. Even on the sentence-level BigVideo benchmark, ViDove remained highly competitive, demonstrating its robustness across different input scales.
Ablation studies highlighted the Proofreader Agent as the most critical component for maintaining quality and terminology consistency. Visual cues were also found to be essential for resolving textual ambiguities that audio alone could not clarify.
Conclusion
ViDove demonstrates that a "human-in-the-loop" agentic workflow, combined with multimodal context and RAG-augmented memory, significantly enhances video translation quality. While absolute BLEU scores indicate that fully automated high-quality subtitling remains a challenge, ViDove provides a robust framework for future development. Future research will explore more tightly integrated multimodal models to process audio and visual signals natively within the reasoning loop.
Summary updated