Abstract
While general-purpose video generation models have achieved significant milestones in photorealistic content, animation remains a challenging frontier due to its unique artistic rules and data scarcity. We introduce AniSora, a comprehensive ecosystem designed specifically for animation. AniSora comprises a massive 10-million-clip dataset, a controllable Diffusion Transformer (DiT) architecture, and a specialized evaluation benchmark. By employing a novel spatiotemporal masking module, AniSora enables precise control over animation tasks such as image-to-video generation and localized motion, significantly outperforming existing state-of-the-art models in character consistency and visual smoothness.
Introduction
Animation is not merely "low-resolution reality"; it follows distinct artistic conventions, features exaggerated motions, and often intentionally violates physical laws. Current general-purpose models like Sora and CogVideoX frequently struggle to maintain the stylistic integrity and character consistency required for high-quality animation.
Researchers from Bilibili identified three primary hurdles in the field: the scarcity of high-quality, large-scale animation datasets; the lack of fine-grained control mechanisms for animators (e.g., keyframe interpolation); and the inadequacy of standard evaluation metrics like VBench, which are optimized for photorealism rather than artistic appeal.
The AniSora Framework
The core of AniSora is a Diffusion Transformer (DiT) architecture utilizing a 3D Causal VAE for efficient data compression. The technical centerpiece is the Spatiotemporal Mask Module, which treats various generation tasks—such as Image-to-Video (I2V), keyframe interpolation, and localized animation—as unified masking problems.
By accepting a "guide sequence" and a "mask sequence" alongside text prompts, the model can:
- Interpolate Keyframes: Generate smooth transitions between provided start and end frames.
- Animate Localized Regions: Use spatial masks to animate specific elements (e.g., a character's mouth) while keeping the background static.
- Maintain Style: Generalize across diverse artistic styles through multi-task learning that incorporates static image generation.
Data Processing and Benchmarking
To power this model, the authors curated the AniSora Dataset, processing 1 million long-form animation videos to extract over 10 million high-quality text-video pairs. The pipeline involved scene detection, OCR filtering to remove subtitles, and aesthetic scoring. Captions were generated using a vision-language model (Qwen-VL2) fine-tuned specifically on animation data.
Furthermore, the team established a manual evaluation benchmark of 948 videos covering 100 common actions. They discovered that traditional metrics often fail to align with human judgment in animation; for instance, high-motion sequences might be unfairly penalized for lack of "smoothness" by standard tools. To address this, they developed regression-based metrics trained on human-annotated animation data.
Results and Evaluation
AniSora was benchmarked against leading models including Vidu-1.5, CogVideoX, and OpenSora. Quantitative results show a significant lead in specialized animation metrics:
- Character Consistency: AniSora scored 94.88, outperforming the next best model, MiniMax (89.47).
- Visual Smoothness: AniSora achieved 71.88, nearly doubling the score of CogVideoX (39.91).
Human preference evaluations confirmed these findings, with AniSora consistently ranking highest in overall quality and artistic style maintenance. The researchers noted that while 3D animation is more easily learned due to its engine-based physical consistency, AniSora shows remarkable progress in the more challenging 2D hand-drawn domain.
Conclusion
AniSora represents a significant step toward professional-grade animation generation. By unifying data, architecture, and evaluation specifically for the animation medium, it provides a robust foundation for future creative tools. Future work will focus on integrating Reinforcement Learning (RLHF) to further refine aesthetic alignment and scaling to resolutions beyond 720P to meet industry production standards.
Summary updated