Exploring the Frontiers of Animation Video Generation in the Sora Era: Method, Dataset and Benchmark

Yudong Jiang, Baohan Xu, Siqian Yang, Mingyu Ying, Jing Liu, Chao Xu, Siqi Wang, Yidi Wu, Bingwen Zhu, Yue Zhang, Jinlong Hou, Huyang SunIJCAI 2025 · DOI

Abstract

While general-purpose video generation models have achieved significant milestones in photorealistic content, animation remains a challenging frontier due to its unique artistic rules and data scarcity. We introduce AniSora, a comprehensive ecosystem designed specifically for animation. AniSora comprises a massive 10-million-clip dataset, a controllable Diffusion Transformer (DiT) architecture, and a specialized evaluation benchmark. By employing a novel spatiotemporal masking module, AniSora enables precise control over animation tasks such as image-to-video generation and localized motion, significantly outperforming existing state-of-the-art models in character consistency and visual smoothness.

Introduction

Animation is not merely "low-resolution reality"; it follows distinct artistic conventions, features exaggerated motions, and often intentionally violates physical laws. Current general-purpose models like Sora and CogVideoX frequently struggle to maintain the stylistic integrity and character consistency required for high-quality animation.

Researchers from Bilibili identified three primary hurdles in the field: the scarcity of high-quality, large-scale animation datasets; the lack of fine-grained control mechanisms for animators (e.g., keyframe interpolation); and the inadequacy of standard evaluation metrics like VBench, which are optimized for photorealism rather than artistic appeal.

The AniSora ecosystem: integrating data processing, controllable video generation, and quantitative evaluation.

The AniSora Framework

The core of AniSora is a Diffusion Transformer (DiT) architecture utilizing a 3D Causal VAE for efficient data compression. The technical centerpiece is the Spatiotemporal Mask Module, which treats various generation tasks—such as Image-to-Video (I2V), keyframe interpolation, and localized animation—as unified masking problems.

By accepting a "guide sequence" and a "mask sequence" alongside text prompts, the model can:

  • Interpolate Keyframes: Generate smooth transitions between provided start and end frames.
  • Animate Localized Regions: Use spatial masks to animate specific elements (e.g., a character's mouth) while keeping the background static.
  • Maintain Style: Generalize across diverse artistic styles through multi-task learning that incorporates static image generation.

Data Processing and Benchmarking

To power this model, the authors curated the AniSora Dataset, processing 1 million long-form animation videos to extract over 10 million high-quality text-video pairs. The pipeline involved scene detection, OCR filtering to remove subtitles, and aesthetic scoring. Captions were generated using a vision-language model (Qwen-VL2) fine-tuned specifically on animation data.

Furthermore, the team established a manual evaluation benchmark of 948 videos covering 100 common actions. They discovered that traditional metrics often fail to align with human judgment in animation; for instance, high-motion sequences might be unfairly penalized for lack of "smoothness" by standard tools. To address this, they developed regression-based metrics trained on human-annotated animation data.

Results and Evaluation

AniSora was benchmarked against leading models including Vidu-1.5, CogVideoX, and OpenSora. Quantitative results show a significant lead in specialized animation metrics:

  • Character Consistency: AniSora scored 94.88, outperforming the next best model, MiniMax (89.47).
  • Visual Smoothness: AniSora achieved 71.88, nearly doubling the score of CogVideoX (39.91).
Human preference evaluation results showing AniSora's superiority across multiple dimensions including motion, quality, and character consistency.

Human preference evaluations confirmed these findings, with AniSora consistently ranking highest in overall quality and artistic style maintenance. The researchers noted that while 3D animation is more easily learned due to its engine-based physical consistency, AniSora shows remarkable progress in the more challenging 2D hand-drawn domain.

Conclusion

AniSora represents a significant step toward professional-grade animation generation. By unifying data, architecture, and evaluation specifically for the animation medium, it provides a robust foundation for future creative tools. Future work will focus on integrating Reinforcement Learning (RLHF) to further refine aesthetic alignment and scaling to resolutions beyond 720P to meet industry production standards.

Summary updated

BibTeX
@inproceedings{jiang2025exploring,
  author = {Yudong Jiang and Baohan Xu and Siqian Yang and Mingyu Ying and Jing Liu and Chao Xu and Siqi Wang and Yidi Wu and Bingwen Zhu and Yue Zhang and Jinlong Hou and Huyang Sun},
  title = {{Exploring the Frontiers of Animation Video Generation in the Sora Era: Method, Dataset and Benchmark}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/141},
}
Figure