TL;DR
This paper introduces a novel deep learning approach for generating realistic speech animation that synchronizes with audio input. Their method uses a sliding window neural network to learn complex mappings from phoneme sequences to facial movements, offering real-time performance, speaker independence, and easy retargeting to various character rigs.
What Problem Does This Paper Address?
Creating realistic and synchronized speech animation for characters in movies and video games is a challenging and often costly process. Traditional methods either rely on expensive performance capture or manual animation by skilled artists, which is difficult to scale and adapt to different languages or characters. Simpler automated methods often produce lower-quality results due to limitations in capturing the subtle nuances of speech, such as coarticulation (how sounds influence each other). The paper aims to bridge this gap by developing an automated, high-fidelity speech animation system that is both efficient and adaptable for production pipelines.
Methods
The core of the proposed approach is a deep learning sliding window regression model. Instead of processing speech frame by frame, the model takes a sequence of phoneme labels as input and predicts a corresponding sequence of facial animation parameters. This sliding window approach is inspired by the observation that coarticulation effects are often localized and can exhibit complex temporal dynamics.
Key aspects of their method include:
- Phoneme-based Input: The system takes phoneme sequences as input, which are obtained by transcribing spoken audio using off-the-shelf speech recognition software. This makes the system speaker and language independent.
- Sliding Window Regression: A deep neural network (DNN) is trained to predict a fixed-length output sequence of animation parameters from a fixed-length input sequence of phonemes. This allows the model to capture local context and coarticulation effects.
- Reference Face Parameterization: The DNN is trained on a single "reference face" model, which is parameterized using an Active Appearance Model (AAM). This simplifies the training data collection and allows the model to focus on learning the general dynamics of speech animation.
- Rig-Space Retargeting: After generating animation parameters for the reference face, a rig-space retargeting method is used to transfer these animations to any target character rig. This involves pre-computing a mapping from the reference face's animation space to the target character's rig parameters.
Datasets
The primary dataset used for training and evaluation is the KB-2k dataset. This dataset contains audio-visual recordings of a single actor reciting 2543 phonetically diverse TIMIT sentences in a neutral tone. The video is high-resolution, and all sentences are manually annotated with Arpabet phonetic codes. The authors note that this dataset was previously used in [Taylor et al. 2012].
Key Insights and Ideas
- Coarticulation is Key: The paper emphasizes that accurately capturing coarticulation effects is crucial for realistic speech animation. Their sliding window approach is specifically designed to learn these complex, context-dependent dynamics.
- Speaker Independence via Phonemes: By using phoneme sequences as input rather than raw audio features, the system achieves speaker independence. This means it can animate speech from any speaker, as long as the audio can be transcribed into phonemes.
- Sliding Window vs. RNNs/LSTMs: The authors argue that a sliding window approach, with its inductive bias towards local temporal curvature, is more suitable for speech animation than recurrent neural networks (RNNs) or Long Short-Term Memory (LSTM) networks, which are designed for longer-range dependencies. They present empirical results supporting this claim.
- Retargeting is Essential for Production: The inclusion of a rig-space retargeting method highlights the practical focus of the paper, aiming to make the generated animations usable on diverse character models in existing animation pipelines.
Key Results
- Quantitative Performance: The proposed approach consistently achieved the lowest mean squared error (MSE) across various metrics (AAM parameter space, shape vertex positions, and appearance pixel intensities) when compared to baselines like Decision Tree Regression, HMM-based Synthesis, and LSTM.
- User Preference Study: Subjective user studies showed that their approach was preferred over baselines and was comparable to the ground truth AAM reference representation. This preference held even when tested on novel speakers and sentences not seen during training.
- Real-time Performance: The system can generate predictions at approximately 1000 video frames per second, making it suitable for real-time applications.
- Generalization: The system demonstrated the ability to generate animation for singing, foreign languages, and different characters, showcasing its generalized nature.
Limitations and Future Work
- Retargeting Errors: A primary limitation is that the animation is generated in the reference face's parameterization, and retargeting to a character can introduce errors. The authors suggest that future work could involve developing data-driven retargeting techniques tailored for automated speech animation.
- Neutral Speech Focus: The current model is trained on neutral speech, requiring artists to add emotional expressions post-generation. Future work could explore training on multi-emotional datasets to directly generate expressive animation.
- Single Speaker Training: The current approach trains on a single speaker. Generalizing to animation for multiple speakers with diverse facial characteristics is a future direction, though it presents a significant data collection challenge.
- Data-Driven Retargeting: The authors suggest exploring data-driven retargeting methods to improve the fidelity of animations transferred to character rigs.
Summary updated