以多層感知器辨識情緒於國台客語料庫 (Use Multilayer Perceptron To Recognize Emotion in Mandarin, Taiwanese and Hakka Database) [In Chinese]

Chia-Hsien Chan, Chia-Ping ChenROCLING 2016

Introduction

Speech Emotion Recognition (SER) is a critical component in developing emotionally intelligent human-computer interfaces. While significant progress has been made using Western datasets like the German EMO-DB, emotional expression is deeply rooted in linguistic and cultural nuances. In the context of Taiwan, where Mandarin, Taiwanese (Hokkien), and Hakka are prevalent, there is a need for localized models. This paper, originally presented at ROCLING 2016, investigates the use of Multilayer Perceptrons (MLPs) to recognize seven distinct emotions across these three languages. The researchers specifically address the challenges of small dataset sizes and speaker variability through architectural choices and normalization techniques.

Methodology

Feature Extraction

The effectiveness of any SER model depends heavily on the input features. The authors utilized a 180-dimensional feature vector derived from 15 "Low-Level Descriptors" (LLDs), including 13 Mel-Frequency Cepstral Coefficients (MFCCs), Pitch, and Zero-Crossing Rate. To capture temporal dynamics, the first-order differences (deltas) of these LLDs were calculated. Finally, six statistical functionals—mean, standard deviation, maximum, minimum, range, and variance—were applied to these 30 descriptors (15 LLDs + 15 deltas), resulting in the final 180-dimensional input.

MLP Architecture

The study compared two neural network configurations implemented in TensorFlow. The first was a single-layer MLP with 35 hidden neurons, and the second was a two-layer MLP with 60 and 15 neurons respectively. To mitigate overfitting, a common issue with small datasets, the researchers employed Dropout layers and Attribute Normalization, scaling all input features to a range between -1 and 1.

Figure 1: The single-layer MLP architecture used for emotion recognition, featuring 180 input dimensions and 35 hidden neurons.

Cross-Speaker Histogram Equalization (CSHE)

A significant hurdle in SER is the interference of speaker-specific vocal characteristics. To isolate the emotional signal, the authors implemented Cross-Speaker Histogram Equalization (CSHE). This technique maps the feature distribution of individual speakers to a reference "virtual speaker" distribution. By normalizing these distributions, the model can focus on the acoustic variations caused by emotion rather than individual identity.

Dataset and Preprocessing

The researchers developed a custom emotional speech corpus modeled after EMO-DB. It includes 30 speakers (10 per language, balanced by gender) expressing seven emotions: Anger, Boredom, Disgust, Fear, Happiness, Sadness, and Neutral. To ensure the reliability of the labels, 10 human evaluators screened the recordings. Only utterances with a human recognition rate exceeding 60% were retained, resulting in a final dataset of 1,261 clips. Validation was performed using the "Leave-One-Speaker-Out" (LOSO) method to test the model's generalizability to unseen voices.

Experimental Results and Discussion

The experiments yielded several key insights regarding model depth and the impact of multi-lingual training data.

Model Depth and Overfitting

Contrary to the trend of "deeper is better," the single-layer MLP consistently outperformed the two-layer version across almost all test scenarios. As shown in the results table, the two-layer model suffered a significant drop in accuracy. This suggests that for a relatively small dataset of ~1,200 samples, the increased complexity of a deeper network leads to overfitting, where the model memorizes training noise rather than learning generalizable emotional patterns.

The Impact of CSHE and Mixed-Language Training

The most significant finding was the synergy between CSHE and mixed-language training. When training on "RAW" (un-normalized) data, adding more languages often decreased performance due to the added linguistic and speaker noise. However, once CSHE was applied, the benefits of the larger, combined dataset became evident.

Figure 5: Comparison of recognition accuracy (%) across different training sets and MLP configurations, highlighting the performance boost from CSHE and multi-lingual data.

As seen in Figure 5, for the single-layer MLP with CSHE, the accuracy improved as more languages were added to the training set. For instance, training on a mix of Mandarin, Taiwanese, and Hakka (國+台+客) yielded the highest overall recognition rates. This indicates that the acoustic features of emotions share significant commonalities across these three languages, allowing the model to leverage the increased data volume effectively.

Language-Specific Performance

The best-performing configuration (Mixed Lang + CSHE) achieved notable improvements over single-language baselines. For example, Hakka recognition rates reached up to 64.6%, a significant jump from the 54.4% baseline. This demonstrates how "low-resource" languages can benefit from being paired with related datasets through effective normalization.

Conclusion

This research demonstrates that a relatively simple MLP architecture, when paired with robust speaker normalization (CSHE), can effectively recognize emotions in a multi-lingual context. The findings highlight that linguistic diversity can be an asset rather than a hindrance if speaker variability is properly managed. Future work will focus on expanding the corpus and investigating why certain emotions, such as Happiness and Sadness, remain acoustically similar and difficult for the model to distinguish in these specific languages.

Summary updated

BibTeX
@inproceedings{chan2016use,
  author = {Chia-Hsien Chan and Chia-Ping Chen},
  title = {{以多層感知器辨識情緒於國台客語料庫 (Use Multilayer Perceptron To Recognize Emotion in Mandarin, Taiwanese and Hakka Database) [In Chinese]}},
  booktitle = {ROCLING},
  year = {2016},
}
Figure