In traditional 2D animation production, creating the illusion of life is an incredibly grueling, manual process. Even after lead animators draw the primary "keyframes" that define an action, two massive production bottlenecks remain: inbetweening (drawing the thousands of intermediate frames that bridge the gaps between keyframes) and colorization (the "paint bucket" phase where artists manually fill every line-enclosed region with flat colors).
A single character design can easily contain dozens of distinct closed segments per frame. Multiplying this by 24 frames per second of video yields an overwhelming amount of repetitive manual labor.
To eliminate this bottleneck, researchers have developed automated pipelines that propagate color and motion from sparse human inputs—such as a single colored master frame, a character design sheet, or simple text tags—across entire sequences of uncolored line art.
Why Automation is Hard: The Geometric Challenge
Automating the paint-bucket tool seems simple at first glance: why not just track the pixels or regions from one frame to the next? In practice, standard computer vision tracking fails because animation graphics are dynamic and non-linear.
When a character moves, three major complications occur:
- Deformation: Character features stretch, squash, and bend.
- Topological Changes: As a character turns, line-enclosed regions frequently merge together or split apart (e.g., a jacket folding over itself).
- Occlusions: Elements completely disappear behind other objects (e.g., a hand moving behind a character's back) and reappear later.
Early computer-assisted approaches attempted to solve this by treating frames as structural maps. For instance, Noma et al. represented animation frames as graphs and applied Belief Propagation to match regions between a master frame and subsequent frames. This allowed the system to track and propagate colors even when regions underwent significant topological merging or splitting.
[Master Frame] ---> (Graph Representation) ---> [Belief Propagation] ---> [Target Frame]
(Colored) (Matches Regions) (Colorized)
However, strict one-to-one region matching inevitably breaks during severe occlusions or complex character deformations. To overcome this, Zhou et al. shifted the paradigm from exact region matching to inclusion matching. Instead of forcing the system to find a perfect counterpart for every single line-enclosed segment, their pipeline learns how segments are included within broader reference regions. This allows the algorithm to correctly color a partially occluded or heavily deformed region by understanding its semantic context within the larger shape.
Generative Colorization and Spatial Entanglement
With the rise of deep learning, researchers turned to generative models—specifically Conditional Generative Adversarial Networks (cGANs) and Conditional Diffusion Models—to synthesize colors directly rather than tracking them explicitly.
Instead of relying on dense frame-by-frame tracking, these models use sparse guidance. Thasarathan et al. demonstrated a cGAN pipeline that utilizes temporal conditioning (feeding the previous colorized frame into the network) alongside sparse artist color hints to maintain temporal consistency. To lower user effort even further, Kim et al. introduced Tag2Pix, which completely replaces manual color brushstrokes with simple text tags (e.g., "blue hair", "red eyes") to guide the generation process. More recently, Carrillo et al. introduced Diffusart, leveraging conditional diffusion models to achieve significantly higher structural accuracy and boundary respect than traditional GANs.
The Problem of Spatial Entanglement
When a generative model uses a reference image (such as a character design sheet) to colorize a target sketch, it frequently suffers from a failure mode known as spatial entanglement. Because deep neural networks are highly sensitive to spatial structures, the model often accidentally copies the pose, layout, or geometry of the reference image onto the target sketch, rather than just copying its color palette and style.
Naive Diffusion:
[Reference Image (Pose A)] + [Target Sketch (Pose B)] ---> Model copies Pose A's geometry onto Sketch B (Artifacts)
Disentangled Diffusion:
[Reference + Latent Noise] + [Target Sketch (Pose B)] ---> Model extracts style only ---> Perfect Colorization
To solve this, Yan et al. developed a two-stage training framework for latent diffusion models that injects controlled noise into the reference image embeddings. By degrading the high-frequency spatial details of the reference image during training, the model is stripped of its ability to rely on the reference's exact layout. It is forced instead to extract abstract semantic style and color distributions, resulting in high-fidelity colorization that strictly respects the structural lines of the target sketch.
Unifying Motion and Color: Post-Keyframing
The latest frontier in animation automation moves away from treating inbetweening and colorization as separate, isolated steps. Historically, an production pipeline required generating a complete sequence of black-and-white line art inbetweens before any color could be applied.
Modern architectures collapse these steps into a unified process often called generative post-keyframing.
A prime example is ToonComposer, a framework developed by Li that adapts large video foundation models using sparse sketch injection and spatial adapters. Instead of requiring a dense sketch for every single intermediate frame, the model takes a single colored reference frame and just a few sparse, high-level sketches. It then simultaneously interpolates the missing motion and synthesizes the color, generating a fully rendered, temporally consistent video sequence.
To ensure that these generated frames look natural and fluid, data-driven video models are increasingly paired with physical constraints. Xie et al. introduced PhysAnimator, which couples 2D deformable body physics simulations with video diffusion models. By using physics to guide the underlying motion of the sketches and generative AI to "re-render" the final colored frames, the system ensures that secondary motions—such as the realistic sway of hair or clothing—remain both physically plausible and stylistically consistent across the entire animation.
Key Foundational Work
Understanding of Stories for Animation (1988) — contributed 2 concept elements to this direction.
Automated lip‐sync: Background and techniques (1991) — contributed 1 concept elements to this direction.