TL;DR
This paper presents a novel analysis of acted French speech, combining acoustic and visual data from a professional actor expressing six emotions. The goal is to identify key characteristics for more believable audiovisual speech synthesis, finding that while some expressions are subtle, others are well-defined in both modalities, particularly joy, surprise, and anger.
What Problem Does This Paper Address?
Creating realistic and expressive virtual talking heads for speech synthesis is a significant challenge. While basic speech synthesis is well-established, conveying emotions convincingly through both sound and facial animation remains difficult. Existing research often focuses on spontaneous speech or perception, but for synthesis, we need to understand how to generate expressive cues. This paper tackles this by analyzing acted speech, which, like synthesized speech, involves an intentional effort to convey emotion, often with some exaggeration, making it a valuable model for synthesis. The authors aim to identify the specific acoustic and visual features that characterize different emotions in acted speech, which can then inform the development of more expressive audiovisual speech synthesis systems.
Methods
The researchers employed a motion capture system with 4 Vicon cameras to record the 3D positions of reflective markers placed on an actor's face. This captured detailed facial movements at 100 Hz. Simultaneously, audio data was recorded using a unidirectional microphone. Synchronization between audio and visual streams was achieved using an in-house electronic device.
For data analysis, they first performed dimensionality reduction on the high-dimensional visual data using Principal Component Analysis (PCA). This helped identify the major directions of facial deformation for each emotion. Acoustic analysis focused on global features of the entire sentence, including pitch (F0) and energy statistics (mean, median, standard deviation, min, max, range), as well as speech rate. Features like jitter and shimmer were also computed but found to be less discriminative for this corpus.
Datasets
The paper introduces a new dataset created by recording a semi-professional French actor uttering 10 sentences for 7 different emotions (neutral, joy, surprise, fear, anger, sadness, disappointment). The actor used an "exercise in style" technique to dissociate semantics from emotion, allowing for a focused analysis of expressive cues. This dataset is a case study, with only one actor's performance analyzed.
Key Insights and Ideas
- Acted speech as a synthesis model: The authors argue that acted speech, with its intentional exaggeration, serves as a good model for generating expressive cues in synthetic speech, as both involve an effort to convey emotion.
- Multimodal expression: Certain emotions are clearly characterized by both acoustic and visual features, suggesting that multimodal analysis is crucial for expressive synthesis.
- Dominant facial movements: For emotions like joy, surprise, and anger, the entire face shows significant movement, with eyebrows and forehead being particularly active.
- Subtlety in some emotions: While some emotions are strongly expressed, others, like fear, show less pronounced facial variations, and their acoustic features can be harder to distinguish.
- Eyebrow movement significance: Eyebrow movement appears to be a key visual cue for distinguishing several emotions, though not consistently for all.
- Acoustic intensity: Acted speech generally exhibits higher acoustic intensity (F0 and energy) compared to neutral speech, suggesting a heightened level of affect or activation.
Key Results
- Visual Analysis: PCA revealed that joy, surprise, and anger showed the highest variance in facial movements, particularly in the upper face (eyebrows, forehead). Sadness and disappointment showed more moderate variations. Fear exhibited the lowest facial variation among the emotional states.
- Acoustic Analysis:
- F0: Anger, joy, fear, and surprise had higher mean F0 values than neutral. Sadness also showed a relatively high mean and maximum F0.
- Intensity: Joy and anger showed significantly higher intensity levels than other emotions.
- Speech Rate: Anger, joy, fear, and surprise were spoken at a faster rate than neutral speech. Sadness and disappointment were closer to the neutral rate.
- Discrimination: While some emotions were well-discriminated by specific features (e.g., joy/anger by intensity, joy/surprise/anger by facial movement), no single feature universally distinguished all emotions.
Limitations and Future Work
- Single Actor: The study is a case study involving only one actor, limiting the generalizability of the findings. Future work plans to include more actors to extract common feature patterns.
- Head Movement: Head movement was excluded from the analysis in this study but is identified as a feature that could help characterize emotions and will be considered in future work.
- Timing Analysis: The timing of visual movements in relation to acoustic events was not investigated but is a promising avenue for future research.
- Synthesis Integration: The authors suggest that combining acoustic and visual features into a single vector for unit-selection audiovisual synthesis could improve emotion perception.
Summary updated