ANIM-400K: A Large-Scale Dataset for Automated End-To-End Dubbing of Video

Kevin Cai, Chonghua Liu, David M. ChanICASSP 2024

Automated video dubbing represents a significant challenge in the field of multimedia accessibility. Unlike subtitling, which relies on Automatic Speech Recognition (ASR) and Machine Translation (MT), dubbing requires a synchronized harmony between translated audio and visual cues. A successful dub must maintain isochrony (timing), prosody (emotional tone), and lip-syncing (visual alignment). Historically, progress in this area has been stifled by a "data desert"—a lack of large-scale, high-quality datasets that pair multi-lingual audio with the original video.

To address this, researchers from UC Berkeley have introduced Anim-400K, a massive dataset comprising over 425,000 aligned Japanese-English video clips. Totaling 763 hours of content, Anim-400K is over 40 times larger than previous benchmarks, providing the necessary scale for training modern end-to-end neural networks.

The Anim-400K Dataset

The dataset is structured hierarchically, spanning 190 shows and approximately 3,500 episodes. This structure allows researchers to leverage context at multiple levels, from broad show-level metadata to granular clip-level timing.

Overview of the Anim-400K dataset hierarchy and scale.

Anim-400K includes raw video, aligned Japanese and English audio tracks, ASR transcripts for both languages, and English subtitles for the Japanese audio. Furthermore, the dataset is enriched with metadata including genres, themes, character biographies, and user ratings, making it a versatile resource for tasks beyond dubbing, such as video summarization and character identification.

Methodology

The authors employed a "top-down" alignment strategy to overcome the limitations of word-for-word translation in professional dubbing. Professional dubs often take creative liberties to ensure the dialogue fits the character's mouth movements and the scene's emotional weight, making strict linguistic alignment difficult.

  1. Alignment: Using AWS Transcribe, the team generated ASR transcripts for both audio tracks. They then recursively merged overlapping segments based on timestamps. Segments were considered aligned if their endpoints were within a 125ms threshold.
  2. Speaker Diarization: The PyAnnote tool was used to identify single-speaker versus multi-speaker clips, a critical distinction for training models to handle overlapping dialogue.
  3. Metadata Integration: By joining the video data with external anime databases, the researchers added layers of semantic information, including 1,800+ character bios and 7,500+ images.

Comparative Analysis

Prior to Anim-400K, the Heroes corpus was the primary resource for dubbing research, containing only 5 hours of content. Other datasets like MuST-C and MuST-Cinema, while large, often lack either the target-language audio or the source-language video required for end-to-end dubbing evaluation.

Comparison of Anim-400K with existing dubbing and translation datasets.

As shown in the table above, Anim-400K significantly outperforms existing benchmarks in terms of hours, clip count, and the availability of both source and target video/audio components. It provides 1.69 million Japanese sentences and 1.20 million English sentences, offering a linguistically rich environment with higher information density per clip compared to Heroes.

Key Insights and Limitations

The "top-down" approach captures vital "performance content"—such as gasps, pauses, and non-speech utterances—that are often missing from traditional transcripts but are essential for realistic dubbing. However, the authors note that roughly 26% of clips contain multiple speakers, representing a significant hurdle for current AI systems that struggle with concurrent voice isolation.

While Anim-400K is a landmark contribution, it is currently limited to the anime domain. Models trained on this data may require fine-tuning to generalize to live-action films or educational content. Additionally, the authors emphasize the ethical importance of using such technology to expand accessibility for underserved content rather than replacing human creative professionals. Future work will focus on improving "cultural competence" in automated translation to ensure that dubs remain respectful of their source material's nuances.

Conclusion

Anim-400K provides the first large-scale foundation for end-to-end video dubbing. By bridging the gap between translation and visual synchronization, it opens new avenues for making global media more accessible to audiences worldwide.

Summary updated

BibTeX
@inproceedings{cai2024anim,
  author = {Kevin Cai and Chonghua Liu and David M. Chan},
  title = {{ANIM-400K: A Large-Scale Dataset for Automated End-To-End Dubbing of Video}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
}
Figure