Abstract
Human emotion recognition is a critical component of biometrics and information forensics. While most research focuses on single RGB views, this paper introduces two new datasets: a RGBD video-emotion dataset and a RGBD face-emotion dataset. We propose a supervised nonlinear Multi-view Laplacian Eigenmaps (MvLE) approach combined with a Multi-hidden-layer Out-of-sample Network (MHON) for robust RGB-D emotion recognition. Central to MvLE is a novel distance metric called Bag-of-Neighbors (BON), which aligns distributions between disparate feature spaces. Our results demonstrate that MvLE effectively handles varying feature sizes and sample counts, significantly outperforming state-of-the-art methods like CCA and MvDA.
Introduction
Recognizing human emotions is essential for advanced human-computer interaction, yet standard RGB-based systems often struggle with lighting variations and complex spatial dynamics. RGB-D sensors, such as the Microsoft Kinect, offer depth information that can mitigate these issues. However, fusing RGB (color/texture) and Depth (shape/distance) data is challenging because they reside in fundamentally different feature spaces. This paper addresses these challenges by proposing a framework that maps multi-view data into a common, low-dimensional latent space using nonlinear manifold learning.
Methodology
The proposed framework addresses the heterogeneity of RGB and Depth data through a supervised nonlinear manifold learning approach that bridges the gap between different sensor modalities.
Bag-of-Neighbors (BON) Metric
The core innovation of this work is the BON metric. Instead of comparing raw feature vectors, which may have different dimensions or scales, BON characterizes each sample by the class distribution of its $K$-nearest neighbors. This "social circle" approach provides a semantic bridge: if two samples from different views share similar neighborhood distributions (e.g., both are surrounded by "Happy" labels), they are considered close in the latent space, regardless of their original feature differences.
Multi-view Laplacian Eigenmaps (MvLE)
MvLE uses the BON connections to construct a global graph across all views. It then solves an optimization problem to find a low-dimensional embedding where connected points are pulled together. Unlike linear methods like Canonical Correlation Analysis (CCA), MvLE is nonlinear, allowing it to capture the complex manifold structure of emotional data, which the authors found to be highly non-linear in video features like C3D.
Multi-hidden-layer Out-of-sample Network (MHON)
To handle new, unseen data (the out-of-sample problem), the authors developed MHON. This network uses a "guiding layer" that is trained to reproduce the low-dimensional representations found by MvLE. This allows for rapid and accurate classification of test samples without re-running the entire manifold learning process for every new input.
Experimental Results
The researchers introduced two new benchmarks to validate their approach: a RGBD Video-Emotion Dataset (4,000+ clips) and a RGBD Face-Emotion Dataset (~1,000 image pairs).
Experiments showed that MvLE significantly outperforms traditional methods. On the video dataset, MvLE achieved approximately 41% accuracy, compared to just 15% for CCA-LDA and 34% for MvDA. On the face dataset, it reached 86% accuracy. The method proved remarkably robust even at very low dimensionalities, maintaining high performance where other multi-view methods typically degrade.
Conclusion
By leveraging the semantic information in local neighborhoods through BON and applying nonlinear manifold learning via MvLE, this research provides a powerful framework for multi-modal emotion recognition. The introduction of new RGB-D datasets further supports the community in developing more sophisticated biometric systems. Future work will focus on reducing the computational complexity of the global graph construction to scale the method for larger datasets.
Summary updated