TL;DR
AnimeInterp is a specialized framework designed to handle the unique challenges of animation video interpolation: textureless color regions and exaggerated, large-scale motions. By combining a segment-guided global matching module with a recurrent flow refinement network, it significantly outperforms general-purpose state-of-the-art models on cartoon data.
What Problem Does This Paper Address?
In the animation industry, frames are often hand-drawn at low rates (e.g., 8 or 12 fps) to save costs. While automated video interpolation could bridge this gap to 24 or 60 fps, standard methods designed for natural video fail on cartoons due to two main factors:
- Lack of Texture: Cartoons consist of smooth color pieces bounded by lines. Standard optical flow algorithms rely on local texture to match pixels between frames; in a flat-colored region, these algorithms get "lost," leading to incorrect motion estimation.
- Exaggerated Motion: Animation often features non-linear and extremely large displacements to convey energy or emotion. These "jumps" frequently exceed the search range or linear assumptions of traditional interpolation models.
Methods
The authors propose AnimeInterp, a coarse-to-fine framework consisting of two primary modules:
1. Segment-Guided Matching (SGM)
To solve the "lack of texture" problem, SGM moves away from pixel-level matching.
- Segmentation: It uses a Laplacian filter and a "trapped-ball" algorithm to partition frames into discrete color segments.
- Global Matching: It extracts deep features (via VGG-19) for each segment. Instead of looking at local pixel neighborhoods, it performs global semantic matching between segments in Frame 0 and Frame 1.
- Coarse Flow: By treating segments as whole units, the model avoids local minima in smooth areas and generates a robust, albeit coarse, piece-wise optical flow.
2. Recurrent Flow Refinement (RFR)
To handle "exaggerated motion" and refine the coarse SGM output, the authors introduce a transformer-inspired recurrent network.
- Confidence Masking: It first learns a pixel-wise confidence map to mask out unreliable matches from the SGM stage.
- Iterative Updates: Inspired by the RAFT architecture, it uses a Convolutional GRU (ConvGRU) to iteratively refine the flow by looking at the correlation between feature maps. This allows the model to gradually "fix" the coarse flow and capture complex deformations.
3. Warping and Synthesis
Finally, the refined bidirectional flows are used with SoftSplat (softmax splatting) and a GridNet synthesis network to generate the final interpolated middle frame.
Datasets
The authors introduce ATD-12K, a large-scale benchmark specifically for animation:
- Scale: 12,000 triplets (Frame 0, Middle, Frame 1) selected from 30 different animation movies.
- Diversity: Covers various styles (Disney, Ghibli, etc.) and over 25 hours of footage.
- Rich Annotations: Triplets are categorized by difficulty (Easy, Medium, Hard) based on motion magnitude and occlusion. They also include Region of Interest (RoI) bounding boxes for moving objects and motion tags (e.g., "scaling," "deformation," "walking").
- Comparison: Statistics show ATD-12K has significantly larger and more diverse motion distributions than natural video datasets like Adobe240.
Key Insights and Ideas
- Segments as Semantic Units: In the absence of texture, the shape and color of a segment are better descriptors than the values of individual pixels. Matching segments globally acts as a powerful prior for motion.
- Coarse-to-Fine Synergy: SGM provides the "big picture" (handling large displacements that would baffle local search), while RFR provides the "fine details" (handling non-linear deformations).
- The Animation Gap: The paper formally demonstrates that animation is not just "low-texture natural video"—its motion patterns are statistically distinct, necessitating domain-specific architectures.
Key Results
AnimeInterp was compared against SOTA methods like DAIN, SoftSplat, and AdaCoF.
- Quantitative: AnimeInterp achieved a PSNR of 29.68 dB on the whole test set, outperforming the runner-up (SoftSplat) by 0.34 dB. On the "Hard" subset (large motions), the improvement was even more pronounced.
- Qualitative: As seen in Figure 5, AnimeInterp successfully restores complex structures (like a character's arm or a fast-moving background) where other methods produce blurry or "broken" artifacts.
- User Study: In a subjective test with ten participants, AnimeInterp's results were preferred over other methods more than 83% of the time.
Limitations and Future Work
- Segmentation Dependence: The SGM module relies on clean color piece segmentation. Extremely complex scenes with heavy gradients or artistic "painterly" styles might challenge the trapped-ball algorithm.
- Scene Transitions: The dataset filters out scene cuts (SSIM < 0.75). The current model is designed for intra-scene interpolation, not for handling abrupt transitions between shots.
- Future Directions: The authors suggest that future work should focus specifically on improving performance in Regions of Interest (RoIs), as these foreground movements are what audiences notice most.
For more details, code, and the ATD-12K dataset, visit the official project page.
Summary updated