TL;DR
This paper introduces ViDove, a novel multimodal translation agent system that leverages visual, audio, and textual inputs to achieve human-like translation quality for long-form video content. By integrating a sophisticated memory system and a multi-agent collaboration framework, ViDove significantly outperforms existing methods and introduces DoveBench, a new benchmark for video subtitling.
What Problem Does This Paper Address?
Current state-of-the-art machine translation (MT) systems, even those enhanced by large language models (LLMs), are largely limited to text-only inputs. This overlooks crucial contextual information present in real-world scenarios, such as videos, where visual cues, audio tone, and speaker emotion significantly influence meaning. Professional human translators naturally integrate these multimodal signals to produce accurate and nuanced translations. Existing multimodal machine translation (MMT) systems often struggle with document-level translation or cannot process entire videos. ViDove aims to bridge this gap by creating an agent that mimics human translators' ability to process and reason over multimodal inputs for enhanced translation quality, particularly for long-form video content like subtitles.
Methods
ViDove is built upon a multi-agent architecture, inspired by human translator workflows. It comprises several specialized agents:
- Vision Agent: Extracts visual cues from video frames to disambiguate text and enhance understanding.
- Auditory Agent: Processes audio streams to extract transcriptions, speaker information, and emotional cues. It leverages SpeechLMs for unified audio cue extraction.
- Translation Agent: Utilizes the extracted multimodal cues and a memory system to generate initial translations.
- Proofreader Agent: Refines the initial translation by checking for grammar, style, terminology, and consistency with previous segments.
- Editor Agent: Performs a final quality check, incorporating multimodal context and user instructions for comprehensive accuracy and coherence.
A key innovation is the Memory System, which includes:
- Short-term Memory (Mˢ): Stores immediate context like translation history, visual cues, and audio cues for the current video chunk.
- Long-term Memory (Mᴸ): Accumulates domain-specific knowledge and general web knowledge to ensure adaptability and consistent performance across diverse tasks and domains.
The system also employs Retrieval-Augmented Generation (RAG) principles, where the memory system acts as a knowledge base for retrieval, and a multi-agent collaboration approach for refinement.
Datasets
- DoveBench: A newly introduced benchmark dataset designed for long-form video automatic subtitling. It comprises approximately 17 hours of video data, with human-annotated Chinese subtitles. The dataset is divided into two categories: CS (23 Counter-Strike videos) and SC2 (27 StarCraft 2 videos).
- BigVideo: Used for evaluating multimodal machine translation (MMT) on sentence-level tasks.
Key Insights and Ideas
- Multimodal Context is Crucial for Accurate Translation: ViDove demonstrates that integrating visual and audio information significantly improves translation quality, especially for nuanced content where text alone is insufficient.
- Memory-Augmented Reasoning Enhances Consistency: The long-short term memory system is vital for maintaining intra-video consistency in terminology and phrasing, as well as for adapting to domain-specific language.
- Multi-Agent Collaboration Mimics Human Workflow: Breaking down the translation process into specialized agents (Translator, Proofreader, Editor) allows for a more thorough and robust refinement process, akin to how human translation teams operate.
- LLMs Alone Struggle with Long-Form Multimodal Tasks: Even powerful single LLMs (like Gemini-2.5-flash and Qwen-2.5-Omni) show significant limitations on DoveBench, highlighting the need for specialized architectures like ViDove for complex, multimodal tasks.
- DoveBench is a Necessary Benchmark: The introduction of DoveBench addresses a gap in evaluating long-form video subtitling, providing a realistic and challenging dataset for future research.
Key Results
ViDove achieved state-of-the-art results on the DoveBench dataset, showing:
- 28.8% improvement in BLEU compared to the strongest baseline (Whisper + DelTA).
- 58.9% improvement in BLEURT.
- 15.5% reduction in SubER (subtitle quality metric).
On the BigVideo dataset, ViDove remained competitive, demonstrating its generalizability beyond the specific long-form subtitling task. The ablation study revealed that removing the domain memory significantly impacted BLEU and SubER, while excluding the proofreader agent led to the sharpest quality drop, underscoring the importance of these components.
Limitations and Future Work
- Modest Absolute Scores: Despite significant improvements, the absolute scores on DoveBench (especially BLEU) remain relatively modest, indicating the inherent difficulty of long-form video automatic subtitling and suggesting room for further advancement.
- Generalizability: While ViDove shows robustness on BigVideo, its primary focus is on long-form video subtitling. Further work could explore its adaptability to other multimodal translation tasks.
- Model Agnosticism: The paper mentions ViDove's design is model-agnostic, allowing flexibility in vision-language backends. Future work could explore the impact of different backend models on performance.
- Human-like Performance: The paper aims to bring translation closer to human-level performance. Continued research could focus on further reducing the gap and achieving even more nuanced and contextually aware translations.
Summary updated