Abstract
Sketch animations offer a powerful medium for visual storytelling, from simple flip-book doodles to professional studio productions. While traditional animation requires teams of skilled artists to draw key frames and in-between frames, existing automation attempts still demand significant artistic effort through precise motion paths or keyframe specification. We present FlipSketch, a system that brings back the magic of flip-book animation -- just draw your idea and describe how you want it to move! Our approach harnesses motion priors from text-to-video diffusion models, adapting them to generate sketch animations through three key innovations: (i) fine-tuning for sketch-style frame generation, (ii) a reference frame mechanism that preserves visual integrity of input sketch through noise refinement, and (iii) a dual-attention composition that enables fluid motion without losing visual consistency. Unlike constrained vector animations, our raster frames support dynamic sketch transformations, capturing the expressive freedom of traditional animation. The result is an intuitive system that makes sketch animation as simple as doodling and describing, while maintaining the artistic essence of hand-drawn animation.
Introduction
Traditional animation is a labor-intensive process requiring "in-betweening," where artists manually draw the frames between key poses. While AI has attempted to automate this, most current tools rely on vector-based techniques. These treat sketches as a collection of paths and control points, essentially moving them like digital puppets. However, vector methods are "stroke-constrained," struggling with scene-level interactions where strokes need to be added or removed, such as liquid filling a cup or a character turning around.
FlipSketch addresses these limitations by providing "flip-book magic": a user provides a single static drawing and a text prompt, and the system handles the complex temporal evolution. By leveraging pre-trained text-to-video (T2V) diffusion models, FlipSketch transforms a static sketch into a fluid, text-guided animation that allows for dynamic transformations while maintaining the visual identity of the original drawing.
Methodology
FlipSketch moves away from vector manipulation, treating sketches as raster images (pixels). This allows the model to "redraw" the subject in every frame, facilitating topological changes. The framework, built on the ModelScope T2V architecture, introduces three main technical innovations:
- Sketch-Style LoRA: The base T2V model is fine-tuned using Low-Rank Adaptation (LoRA) on a synthetic dataset of sketch animations. This teaches the model the specific aesthetic of line drawings while retaining the motion priors from real-world video data.
- Iterative Frame Alignment: To ensure the generated video matches the user's input, DDIM inversion converts the input sketch into a unique noise pattern. During denoising, a specific loss function ($L_{align}$) forces the first frame to match the input sketch's features.
- Dual-Attention Composition: To maintain consistency, the system modifies internal attention mechanisms. By injecting spatial and temporal cross-attention from the reference sketch into subsequent frames, the model preserves the structure and style of the original doodle during motion.
Results and Insights
FlipSketch significantly outperforms previous state-of-the-art methods like Live-Sketch in text-to-video alignment (0.172 vs 0.142). User studies indicate that FlipSketch is rated higher for both faithfulness to the text prompt and consistency with the input sketch compared to general image-to-video models like Stable Video Diffusion (SVD).
Key Insights
- Raster Expressivity: Abandoning rigid vector paths allows for "topological changes," such as adding strokes for pouring water or removing them as objects are occluded.
- Motion-Fidelity Trade-off: There is a fundamental tension between fidelity to the original sketch and dynamic motion. FlipSketch introduces a control parameter ($\lambda$) to balance these aspects.
- Diffusion Signals as Features: Treating the denoising signal as a feature map allows for high-level semantic alignment before fine-grained line placement.
Conclusion and Future Work
FlipSketch successfully automates the creation of expressive sketch animations from single drawings. However, challenges remain, including a style bias toward the training data's "sketchy" aesthetic and occasional geometric "corrections" of abstract doodles in the initial frame. Future work will focus on expanding training diversity to support more artistic styles and improving the handling of highly abstract inputs.
Summary updated