Bimodal information analysis for emotion recognition

Malika Meghjani, Frank P. Ferrie, Gregory Dudek

Abstract

Automatic emotion recognition is a cornerstone of advanced human-computer interaction, yet it remains a significant challenge due to the inherent variability in human expression. This paper proposes a robust bimodal system that integrates facial expression analysis with acoustic features to recognize five universal emotions. By combining visual Gabor filters with global audio statistics and utilizing Support Vector Machines (SVM) for classification, the system achieves a recognition rate of 84%. This performance significantly outperforms unimodal approaches, demonstrating the efficacy of bimodal fusion in handling real-world emotional data.

Introduction

The ability to automatically recognize human emotions is increasingly vital for applications ranging from entertainment to healthcare. In the context of Tele-Health, for instance, monitoring the emotional state of patients—such as those suffering from Chronic Obstructive Pulmonary Disease (COPD)—allows healthcare providers to evaluate the effectiveness of interventions remotely. However, unimodal systems that rely solely on facial expressions or vocal cues often struggle in noisy, real-world environments where data may be occluded or distorted.

A primary challenge in this field is the "temporal problem": emotions are dynamic and fluctuate in intensity over time. Identifying which frames of a video sequence are most representative of an emotion is a difficult task that often requires labor-intensive manual labeling. This research addresses these challenges by developing an efficient bimodal architecture that leverages the complementary nature of audio and visual information.

Methodology

The proposed system utilizes a "naïve" yet computationally efficient bimodal architecture designed for robustness.

Visual Feature Extraction

The visual pipeline begins with face detection using the Viola-Jones algorithm. Once the face is localized, features are extracted using a bank of 20 Gabor filters, spanning five spatial frequencies and four orientations. These filters provide a multi-resolution analysis of facial textures, capturing the subtle changes associated with different expressions. To address the temporal problem, the research evaluates several "key frame" selection methods, including an unsupervised clustering approach, to identify the peak intensity of an expression without manual intervention.

Audio Feature Extraction

To avoid the noise often associated with local dynamic audio features, the system extracts global statistics—such as the mean, maximum, and standard deviation—of pitch and intensity. These are supplemented by Mel-Frequency Cepstrum Coefficients (MFCCs), which provide a compact representation of the power spectrum of the audio signal.

Classification and Fusion

A multi-class Support Vector Machine (SVM) is employed as the primary classifier. Given the high dimensionality of the visual feature set (over 20,000 features), Recursive Feature Elimination (RFE) is used to select the most discriminative subset. The research explores two fusion strategies:

  1. Feature-level Fusion: Concatenating the visual and audio feature vectors into a single input for the SVM.
  2. Score-level Fusion: Combining the probability outputs of separate visual and audio classifiers using the product rule.

Results and Discussion

The system was validated using the eNTERFACE 2005 database and the Belfast Naturalistic Database.

Performance Gains

The results confirm the power of bimodality. The best combined recognition rate reached 84% using score-level fusion and unsupervised key frame clustering. This represents a substantial improvement over unimodal performance, where audio alone achieved only 53% and visual analysis reached approximately 80%. The analysis revealed that different modalities excel at identifying different emotions; for example, sadness and fear are more detectable in audio, while anger and surprise are more distinct visually.

Robustness and Efficiency

One of the most significant findings is that a single, well-chosen key frame representing the peak of an emotion can be nearly as effective for training as an entire temporal sequence. Furthermore, the unsupervised clustering approach for selecting these frames matched the accuracy of manual selection by human experts. When tested on the spontaneous Belfast database, the system maintained an 83% accuracy rate for a two-class problem (Happy vs. Sad), proving its ability to generalize to unconstrained, real-world interactions.

Conclusion

This research demonstrates that a bimodal approach to emotion recognition significantly enhances accuracy and robustness. By efficiently selecting key frames and fusing audio-visual data at the score level, the system provides a viable path for real-time emotional monitoring in clinical settings. Future work will explore the use of Hidden Markov Models (HMMs) to better capture the temporal evolution of expressions and investigate more computationally efficient feature descriptors to reduce the overhead of Gabor filtering.

Summary updated

Abstract

We present a bimodal information analysis system for automatic emotion recognition. Our approach is based on the analysis of video sequences which combines facial expressions observed visually with acoustic features to automatically recognize five universal emotion classes: anger, disgust, happiness, sadness and surprise. We address the challenges posed during the temporal analysis of the bimodal data and introduce a novel technique for combining the best features of instantaneous and temporal based visual recognition systems. We obtain robust appearance-based visual features which we classify instantaneously and aggregate it temporally to improve the recognition rates when compared to single-frame based instantaneous classification. The performance of the system is further boosted by using the complementary audio information for the bimodal emotion recognition. We combine the two modalities at both feature and score level to compare the respective joint emotion recognition rates. The emotions are instantaneously classified using a support vector machine and sequentially aggregated based on their classification probabilities. This approach is validated on a posed audio-visual database and a natural interactive database. The experiments performed on these databases provide encouraging results with the best combined recognition rate being 82%.
BibTeX
@inproceedings{meghjani2009bimodal,
  author = {Malika Meghjani and Frank P. Ferrie and Gregory Dudek},
  title = {{Bimodal information analysis for emotion recognition}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2009},
  doi = {10.1109/wacv.2009.5403035},
}
Figure