Deep Animation Video Interpolation in the Wild

Siyao Li, Shiyu Zhao, Weijiang Yu, Wenxiu Sun, Dimitris N. Metaxas, Chen Change Loy, Ziwei Liu

TL;DR

AnimeInterp is a specialized framework designed to handle the unique challenges of animation video interpolation: textureless color regions and exaggerated, large-scale motions. By combining a segment-guided global matching module with a recurrent flow refinement network, it significantly outperforms general-purpose state-of-the-art models on cartoon data.

What Problem Does This Paper Address?

In the animation industry, frames are often hand-drawn at low rates (e.g., 8 or 12 fps) to save costs. While automated video interpolation could bridge this gap to 24 or 60 fps, standard methods designed for natural video fail on cartoons due to two main factors:

  1. Lack of Texture: Cartoons consist of smooth color pieces bounded by lines. Standard optical flow algorithms rely on local texture to match pixels between frames; in a flat-colored region, these algorithms get "lost," leading to incorrect motion estimation.
  2. Exaggerated Motion: Animation often features non-linear and extremely large displacements to convey energy or emotion. These "jumps" frequently exceed the search range or linear assumptions of traditional interpolation models.
Figure 1: A typical example for animation video interpolation. Our approach is capable of estimating optical flows for large motions correctly and restore the content, while competing methods fail to handle such motions.

Methods

The authors propose AnimeInterp, a coarse-to-fine framework consisting of two primary modules:

1. Segment-Guided Matching (SGM)

To solve the "lack of texture" problem, SGM moves away from pixel-level matching.

  • Segmentation: It uses a Laplacian filter and a "trapped-ball" algorithm to partition frames into discrete color segments.
  • Global Matching: It extracts deep features (via VGG-19) for each segment. Instead of looking at local pixel neighborhoods, it performs global semantic matching between segments in Frame 0 and Frame 1.
  • Coarse Flow: By treating segments as whole units, the model avoids local minima in smooth areas and generates a robust, albeit coarse, piece-wise optical flow.

2. Recurrent Flow Refinement (RFR)

To handle "exaggerated motion" and refine the coarse SGM output, the authors introduce a transformer-inspired recurrent network.

  • Confidence Masking: It first learns a pixel-wise confidence map to mask out unreliable matches from the SGM stage.
  • Iterative Updates: Inspired by the RAFT architecture, it uses a Convolutional GRU (ConvGRU) to iteratively refine the flow by looking at the correlation between feature maps. This allows the model to gradually "fix" the coarse flow and capture complex deformations.

3. Warping and Synthesis

Finally, the refined bidirectional flows are used with SoftSplat (softmax splatting) and a GridNet synthesis network to generate the final interpolated middle frame.

Figure 4: The overall pipeline of AnimeInterp, showing the SGM and RFR modules.

Datasets

The authors introduce ATD-12K, a large-scale benchmark specifically for animation:

  • Scale: 12,000 triplets (Frame 0, Middle, Frame 1) selected from 30 different animation movies.
  • Diversity: Covers various styles (Disney, Ghibli, etc.) and over 25 hours of footage.
  • Rich Annotations: Triplets are categorized by difficulty (Easy, Medium, Hard) based on motion magnitude and occlusion. They also include Region of Interest (RoI) bounding boxes for moving objects and motion tags (e.g., "scaling," "deformation," "walking").
  • Comparison: Statistics show ATD-12K has significantly larger and more diverse motion distributions than natural video datasets like Adobe240.

Key Insights and Ideas

  • Segments as Semantic Units: In the absence of texture, the shape and color of a segment are better descriptors than the values of individual pixels. Matching segments globally acts as a powerful prior for motion.
  • Coarse-to-Fine Synergy: SGM provides the "big picture" (handling large displacements that would baffle local search), while RFR provides the "fine details" (handling non-linear deformations).
  • The Animation Gap: The paper formally demonstrates that animation is not just "low-texture natural video"—its motion patterns are statistically distinct, necessitating domain-specific architectures.

Key Results

AnimeInterp was compared against SOTA methods like DAIN, SoftSplat, and AdaCoF.

  • Quantitative: AnimeInterp achieved a PSNR of 29.68 dB on the whole test set, outperforming the runner-up (SoftSplat) by 0.34 dB. On the "Hard" subset (large motions), the improvement was even more pronounced.
  • Qualitative: As seen in Figure 5, AnimeInterp successfully restores complex structures (like a character's arm or a fast-moving background) where other methods produce blurry or "broken" artifacts.
  • User Study: In a subjective test with ten participants, AnimeInterp's results were preferred over other methods more than 83% of the time.
Figure 5: Qualitative comparison showing AnimeInterp's ability to maintain structural integrity during large displacements.

Limitations and Future Work

  • Segmentation Dependence: The SGM module relies on clean color piece segmentation. Extremely complex scenes with heavy gradients or artistic "painterly" styles might challenge the trapped-ball algorithm.
  • Scene Transitions: The dataset filters out scene cuts (SSIM < 0.75). The current model is designed for intra-scene interpolation, not for handling abrupt transitions between shots.
  • Future Directions: The authors suggest that future work should focus specifically on improving performance in Regions of Interest (RoIs), as these foreground movements are what audiences notice most.

For more details, code, and the ATD-12K dataset, visit the official project page.

Summary updated

Abstract

In the animation industry, cartoon videos are usually produced at low frame rate since hand drawing of such frames is costly and time-consuming. Therefore, it is desirable to develop computational models that can automatically interpolate the in-between animation frames. However, existing video interpolation methods fail to produce satisfying results on animation data. Compared to natural videos, animation videos possess two unique characteristics that make frame interpolation difficult: 1) cartoons comprise lines and smooth color pieces. The smooth areas lack textures and make it difficult to estimate accurate motions on animation videos. 2) cartoons express stories via exaggeration. Some of the motions are non-linear and extremely large. In this work, we formally define and study the animation video interpolation problem for the first time. To address the aforementioned challenges, we propose an effective framework, AnimeInterp, with two dedicated modules in a coarse-to-fine manner. Specifically, 1) Segment-Guided Matching resolves the "lack of textures" challenge by exploiting global matching among color pieces that are piece-wise coherent. 2) Recurrent Flow Refinement resolves the "non-linear and extremely large motion" challenge by recurrent predictions using a transformer-like architecture. To facilitate comprehensive training and evaluations, we build a large-scale animation triplet dataset, ATD-12K, which comprises 12,000 triplets with rich annotations. Extensive experiments demonstrate that our approach outperforms existing state-of-the-art interpolation methods for animation videos. Notably, AnimeInterp shows favorable perceptual quality and robustness for animation scenarios in the wild. The proposed dataset and code are available at https://github.com/lisiyao21/AnimeInterp/.
BibTeX
@inproceedings{li2021deep,
  author = {Siyao Li and Shiyu Zhao and Weijiang Yu and Wenxiu Sun and Dimitris N. Metaxas and Chen Change Loy and Ziwei Liu},
  title = {{Deep Animation Video Interpolation in the Wild}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  doi = {10.1109/cvpr46437.2021.00652},
}
Figure