Talking Head(?) Anime from a Single Image 4: Improved Model and Its Distillation

Pramook KhungurnarXiv 2023 · DOI · arXiv

Introduction

The creation of controllable 2D avatars, a cornerstone of the VTubing industry, has traditionally relied on manual "rigging"—a meticulous process of layering and deforming 2D assets. While automated solutions like Talking Head Anime 3 (THA3) demonstrated the potential for single-image animation, they were hampered by two critical flaws: significant detail loss during head or body rotations and high computational requirements that restricted use to high-end workstation GPUs.

This paper, presented by researchers at pixiv Inc., introduces the fourth iteration of the system. THA4 employs a two-stage "Teacher-Student" framework to achieve both high-fidelity animation and real-time performance on consumer-grade hardware.

The Teacher Model: Enhancing Visual Fidelity

The "Teacher" model serves as the high-quality reference for the system. To address the blurriness and detail erasure common in previous versions, the authors overhauled the architecture of the "body rotator" and "editor" modules. Specifically, they replaced standard convolutional neural network (CNN) backbones with U-Nets featuring integrated attention layers.

Qualitative comparison between the previous THA3 model and the proposed Teacher model, highlighting improved detail preservation.

This architectural shift allows the model to better capture global context and maintain the integrity of fine structures, such as individual hair strands, even during complex rotations. Quantitative evaluations showed a ~30% improvement in perceptual quality (LPIPS) over THA3, with the Teacher model maintaining sharp silhouettes and complex textures that were previously lost.

The Student Model: Real-Time Distillation

While the Teacher model achieves superior quality, its inference speed (~150ms per frame) is insufficient for live interaction. The authors solve this through Knowledge Distillation, training a specialized "Student" model to mimic the Teacher's output for a specific character.

The Student model utilizes a SIREN (Sinusoidal Representation Network), a coordinate-based neural network that excels at representing the smooth signals characteristic of anime art. To optimize for speed without sacrificing resolution, the student generates images through a multi-resolution pipeline (128x128 → 256x256 → 512x512) and a hybrid image formation strategy.

Detailed architecture of the Student model and its image formation process.

As illustrated in the architecture above, the Student model does not merely "paint" pixels. Instead, it predicts an appearance flow to warp the original high-resolution input and a separate RGBA image to fill in disoccluded areas. These components are blended using a predicted alpha map, ensuring that the final output preserves the crispness of the original artwork while filling in gaps realistically.

Results and Performance

The specialized Student model is remarkably lightweight, with a file size of approximately 1.8 MB. This efficiency translates to significant performance gains: on a consumer-grade NVIDIA GTX 1080 Ti, the Student model generates frames in just 22ms (~45 FPS), whereas the Teacher model requires 160ms (~6 FPS) on the same hardware.

This "Specialization" trade-off is a key insight of the paper. While the distillation process (or "baking") requires several tens of hours of GPU time per character, it enables high-fidelity, real-time animation that can run efficiently even within a web browser.

Conclusion and Future Work

THA4 represents a significant step forward in automated 2D avatar animation. By combining the representational power of attention-based U-Nets with the efficiency of distilled, coordinate-based networks, the authors have made high-quality VTubing more accessible.

However, limitations remain. The system is currently restricted to relatively small rotation angles, and the lengthy baking time prevents instantaneous character swapping. Future research will focus on expanding the range of motion and further optimizing the architecture for deployment on mobile devices.

Summary updated

Abstract

We study the problem of creating a character model that can be controlled in real time from a single image of an anime character. A solution to this problem would greatly reduce the cost of creating avatars, computer games, and other interactive applications. Talking Head Anime 3 (THA3) is an open source project that attempts to directly address the problem. It takes as input (1) an image of an anime character's upper body and (2) a 45-dimensional pose vector and outputs a new image of the same character taking the specified pose. The range of possible movements is expressive enough for personal avatars and certain types of game characters. However, the system is too slow to generate animations in real time on common PCs, and its image quality can be improved. In this paper, we improve THA3 in two ways. First, we propose new architectures for constituent networks that rotate the character's head and body based on U-Nets with attention that are widely used in modern generative models. The new architectures consistently yield better image quality than the THA3 baseline. Nevertheless, they also make the whole system much slower: it takes up to 150 milliseconds to generate a frame. Second, we propose a technique to distill the system into a small network (less than 2 MB) that can generate 512x512 animation frames in real time (under 30 FPS) using consumer gaming GPUs while keeping the image quality close to that of the full system. This improvement makes the whole system practical for real-time applications.
BibTeX
@misc{khungurn2023talking,
  author = {Pramook Khungurn},
  title = {{Talking Head(?) Anime from a Single Image 4: Improved Model and Its Distillation}},
  year = {2023},
  eprint = {2311.17409},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2311.17409},
}
Figure