Multi-view Laplacian Eigenmaps Based on Bag-of-Neighbors For RGBD Human Emotion Recognition

Shenglan Liu, Shuai Guo, Hong Qiao, Yang Wang, Bin Wang, Wenbo Luo, Mingming Zhang, Keye Zhang, Bixuan Du

Abstract

Human emotion recognition is a critical component of biometrics and information forensics. While most research focuses on single RGB views, this paper introduces two new datasets: a RGBD video-emotion dataset and a RGBD face-emotion dataset. We propose a supervised nonlinear Multi-view Laplacian Eigenmaps (MvLE) approach combined with a Multi-hidden-layer Out-of-sample Network (MHON) for robust RGB-D emotion recognition. Central to MvLE is a novel distance metric called Bag-of-Neighbors (BON), which aligns distributions between disparate feature spaces. Our results demonstrate that MvLE effectively handles varying feature sizes and sample counts, significantly outperforming state-of-the-art methods like CCA and MvDA.

Introduction

Recognizing human emotions is essential for advanced human-computer interaction, yet standard RGB-based systems often struggle with lighting variations and complex spatial dynamics. RGB-D sensors, such as the Microsoft Kinect, offer depth information that can mitigate these issues. However, fusing RGB (color/texture) and Depth (shape/distance) data is challenging because they reside in fundamentally different feature spaces. This paper addresses these challenges by proposing a framework that maps multi-view data into a common, low-dimensional latent space using nonlinear manifold learning.

Methodology

The proposed framework addresses the heterogeneity of RGB and Depth data through a supervised nonlinear manifold learning approach that bridges the gap between different sensor modalities.

The overall framework of the proposed method, illustrating the training stage using MvLE and BON, and the testing stage using the MHON classifier with its guiding layer.

Bag-of-Neighbors (BON) Metric

The core innovation of this work is the BON metric. Instead of comparing raw feature vectors, which may have different dimensions or scales, BON characterizes each sample by the class distribution of its $K$-nearest neighbors. This "social circle" approach provides a semantic bridge: if two samples from different views share similar neighborhood distributions (e.g., both are surrounded by "Happy" labels), they are considered close in the latent space, regardless of their original feature differences.

The BON mechanism represents samples as histograms of their K-nearest neighbors' class labels, effectively normalizing different feature spaces into a common distribution.

Multi-view Laplacian Eigenmaps (MvLE)

MvLE uses the BON connections to construct a global graph across all views. It then solves an optimization problem to find a low-dimensional embedding where connected points are pulled together. Unlike linear methods like Canonical Correlation Analysis (CCA), MvLE is nonlinear, allowing it to capture the complex manifold structure of emotional data, which the authors found to be highly non-linear in video features like C3D.

Multi-hidden-layer Out-of-sample Network (MHON)

To handle new, unseen data (the out-of-sample problem), the authors developed MHON. This network uses a "guiding layer" that is trained to reproduce the low-dimensional representations found by MvLE. This allows for rapid and accurate classification of test samples without re-running the entire manifold learning process for every new input.

Experimental Results

The researchers introduced two new benchmarks to validate their approach: a RGBD Video-Emotion Dataset (4,000+ clips) and a RGBD Face-Emotion Dataset (~1,000 image pairs).

Experiments showed that MvLE significantly outperforms traditional methods. On the video dataset, MvLE achieved approximately 41% accuracy, compared to just 15% for CCA-LDA and 34% for MvDA. On the face dataset, it reached 86% accuracy. The method proved remarkably robust even at very low dimensionalities, maintaining high performance where other multi-view methods typically degrade.

Conclusion

By leveraging the semantic information in local neighborhoods through BON and applying nonlinear manifold learning via MvLE, this research provides a powerful framework for multi-modal emotion recognition. The introduction of new RGB-D datasets further supports the community in developing more sophisticated biometric systems. Future work will focus on reducing the computational complexity of the global graph construction to scale the method for larger datasets.

Summary updated

Abstract

Human emotion recognition is an important direction in the field of biometric and information forensics. However, most existing human emotion research are based on the single RGB view. In this paper, we introduce a RGBD video-emotion dataset and a RGBD face-emotion dataset for research. To our best knowledge, this may be the first RGBD video-emotion dataset. We propose a new supervised nonlinear multi-view laplacian eigenmaps (MvLE) approach and a multihidden-layer out-of-sample network (MHON) for RGB-D humanemotion recognition. To get better representations of RGB view and depth view, MvLE is used to map the training set of both views from original space into the common subspace. As RGB view and depth view lie in different spaces, a new distance metric bag of neighbors (BON) used in MvLE can get the similar distributions of the two views. Finally, MHON is used to get the low-dimensional representations of test data and predict their labels. MvLE can deal with the cases that RGB view and depth view have different size of features, even different number of samples and classes. And our methods can be easily extended to more than two views. The experiment results indicate the effectiveness of our methods over some state-of-art methods.
BibTeX
@misc{liu2018multi,
  author = {Shenglan Liu and Shuai Guo and Hong Qiao and Yang Wang and Bin Wang and Wenbo Luo and Mingming Zhang and Keye Zhang and Bixuan Du},
  title = {{Multi-view Laplacian Eigenmaps Based on Bag-of-Neighbors For RGBD Human Emotion Recognition}},
  year = {2018},
  eprint = {1811.03478},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1811.03478},
}

Loading related content...

Figure