Contributions of audio and visual modalities to perception of Mandarin Chinese emotions in valence-arousal space

Yongwei Li, Aijun Li, Jianhua Tao, Feng Li, Donna Erickson, Masato AkagiNippon 2024 · DOI

TL;DR

This study explores how we combine what we hear and what we see to decode emotions in Mandarin Chinese. By mapping perceptions onto the "valence-arousal" space, researchers found that while audio cues (like pitch) drive our sense of intensity, visual cues (facial expressions) are the essential tie-breakers for distinguishing between positive and negative emotions, such as happiness versus anger.

What Problem Does This Paper Address?

Most research into emotion recognition focuses on "categorical" labels—is the speaker happy, sad, or angry? However, human emotion is a spectrum. Researchers often use a dimensional approach called the Valence-Arousal (V-A) space:

  • Valence: How positive or negative an emotion is (e.g., happy vs. sad).
  • Arousal: How calm or excited the emotion is (e.g., content vs. ecstatic).

While this has been studied extensively in Western languages, Mandarin Chinese presents a unique challenge because it is a tonal language. In Mandarin, pitch ($F_0$) is used to distinguish word meanings, which might constrain how much "room" there is to use pitch for emotional expression. This paper investigates the specific contributions of audio and visual modalities to V-A perception in native Mandarin speakers, filling a gap in our understanding of multimodal communication in tonal languages.

Methods

The researchers conducted three psychoacoustic experiments with 20 native Chinese-speaking participants. They used a set of 70 stimuli: 10 sentences recorded with seven different intended emotions (happiness, sadness, anger, disgust, fear, surprise, and neutral) performed by a professional actress.

The experiments were structured as follows:

  1. Experiment I (Audio-only): Participants rated the valence and arousal of speech clips without seeing the speaker.
  2. Experiment II (Video-only): Participants rated the emotions based on silent video clips of the speaker's face.
  3. Experiment III (Audio-Video): Participants rated the combined multimodal stimuli.

To analyze the "why" behind the ratings, the team used STRAIGHT (a high-quality speech manipulation tool) to extract fundamental frequency ($F_0$) from the audio and the Facial Action Coding System (FACS) to identify specific "Action Units" (AUs)—the building blocks of facial expressions—in the video.

Datasets

The study utilized a controlled, high-quality recording of a professional female actress from the Beijing Film Academy. While not a massive "in-the-wild" dataset, this allowed for precise control over the intended emotional content. The stimuli were validated in a previous study to ensure the intended emotions were clearly recognizable.

Key Insights and Ideas

  • The "Anger-Happiness" Paradox: In the audio-only trials, anger and happiness often ended up in similar spots in the V-A space because both involve high arousal and relatively high pitch. The "aha moment" here is that video is the primary differentiator; facial expressions are what allow us to correctly assign positive valence to happiness and negative valence to anger.
  • Audio as an Intensity Amplifier: The researchers found that compared to video, audio tends to polarize ratings. It makes low-arousal emotions (like sadness) feel even "lower" and high-arousal emotions (like surprise) feel even "higher."
  • Pitch is the Arousal Engine: There was a strong linear correlation ($R = 0.67$) between the fundamental frequency ($F_0$) of the voice and the perceived arousal. If the pitch goes up, we perceive the speaker as more excited, regardless of the specific emotion.
  • Multimodal Integration: When both audio and video are present, the brain doesn't just pick one. The ratings for the combined (AV) stimuli often occupied a middle ground between the audio-only and video-only scores, suggesting a complex integration process where the brain "averages" or weights the cues.

Key Results

  • Statistical Significance: ANOVA tests confirmed that the modality (A, V, or AV) significantly impacted how emotions were perceived in both the valence and arousal dimensions ($p < 0.001$).
  • Visual Correlation: Specific facial movements were highly correlated with valence. For example, "Lip corner puller" (AU12, associated with smiling) had a high correlation with positive valence, while "Lid tightener" (AU7) correlated with negative valence and higher arousal.
  • Arousal Dominance: Audio cues were found to be more influential in determining the arousal level, while visual cues were more influential in determining valence (the "positivity" or "negativity" of the emotion).

Limitations and Future Work

The authors acknowledge a few key limitations:

  • Single Speaker: The stimuli were based on a single professional female speaker. Individual differences in how people express emotion (individuality) could affect the results.
  • Controlled Environment: The recordings were made in a sound-proof room. Real-world "noisy" environments might change how we weight audio versus visual cues.
  • Cultural Specificity: While this study focuses on Mandarin Chinese, the authors suggest that future work should compare these results across different cultures to see if the weighting of audio vs. visual cues is a universal human trait or culturally learned.

This research provides a foundational map for developers working on affective computing and human-computer interaction, particularly for systems designed to interact with Mandarin speakers.

Summary updated

Abstract

Emotions are usually perceived by multimodal cues for human communications; in recent years, emotions have been studied from the perspective of dimensional approaches. Investigation of audio and video cues to emotion perception in terms of categories of emotion has been relatively extensively conducted, but the contribution of audio and video cues to emotion perception in dimensional space is relatively under-investigated, especially in Mandarin Chinese. In this present study, three psychoacoustic experiments were conducted to investigate the contributions of audio, visual, and audio-visual modalities to emotional perception in the valence and arousal space. Audio-only, video-only, and audio-video modalities were presented to native Chinese subjects with normal hearing and vision for perceptual ratings of emotion in the valence and arousal dimensions. Results suggested that (1) different modalities contribute differently to perceiving valence and arousal dimensions; (2) compared to video-only modality, audio-only modality generally decreases arousal and valence at lower levels, and increases arousal and valence at higher levels; (3) the video-only modality plays an important role in separating anger and happiness emotions in the valence space.
BibTeX
@inproceedings{li2024contributions,
  author = {Yongwei Li and Aijun Li and Jianhua Tao and Feng Li and Donna Erickson and Masato Akagi},
  title = {{Contributions of audio and visual modalities to perception of Mandarin Chinese emotions in valence-arousal space}},
  booktitle = {Nippon},
  year = {2024},
  doi = {10.1250/ast.e24.41},
}
Figure