Singular Learning of Deep Multilayer Perceptrons for EEG-Based Emotion Recognition

Weili Guo, Guangyu Li, Jianfeng Lu

Introduction

Training deep neural networks for EEG-based emotion recognition often encounters frustrating "plateaus" where progress nearly stops. This research identifies specific mathematical "singularities" in the network's parameter space—specifically zero weights and overlapping neurons—as the primary cause of these delays. Using the benchmark SEED dataset, the authors demonstrate how these geometric properties of the loss landscape impact training efficiency and model performance.

Problem Statement: The Plateau Phenomenon

Deep learning is a standard tool for decoding human emotions from electroencephalograph (EEG) signals. However, researchers frequently face the "plateau phenomenon," where the loss function stops decreasing for extended periods before a sudden drop. This is not merely a hardware limitation but a fundamental issue with the "geometry" of the neural network. In regions called singularities, the Fisher information matrix becomes singular, causing standard gradient descent algorithms to lose direction and stall. This paper bridges the gap between theoretical singular learning theory and practical EEG applications.

Methods and Architecture

The study focuses on Deep Multilayer Perceptrons (MLPs) as a representative model for complex architectures.

Architecture of the deep Multilayer Perceptron (MLP) used in the study.

Theoretical Analysis

The authors define two primary types of singularities in a network with two hidden layers:

  • Zero Weight Singularity ($R_1$): Occurs when a weight connecting a hidden unit to the output layer becomes zero, effectively removing that path from the network.
  • Overlap Singularity ($R_2$): Occurs when two hidden units have identical weights, making them redundant.

Dataset and Preprocessing

The SEED dataset (SJTU Emotion EEG Dataset) was used, featuring EEG signals from 15 subjects watching clips designed to elicit positive, neutral, or negative emotions. The 62-channel data was divided into five frequency bands. Principal Component Analysis (PCA) reduced the input to a 25-dimension vector for a 25-8-8-1 MLP architecture.

Key Insights and Results

Numerical simulations tracked the "learning trajectories" of weights during training to observe interactions with singular regions.

Training dynamics showing (A) training error, (B) weight trajectories, and (C) classification accuracy, highlighting the plateau phenomenon.

Plateaus as Saddle Points

The results indicate that plateaus caused by zero weight singularities are saddle points rather than local minima. The network eventually escapes, but the path is narrow and time-consuming.

Impact of Singularities

The study compared four cases to quantify the impact:

  • Case 1 (Fast Convergence): 8,000 iterations, 94.8% training accuracy.
  • Case 2 (Zero Weight Singularity): 8,000 iterations, 90.1% training accuracy (trapped in plateau).
  • Case 3 (Extended Training): 15,000 iterations, 94.4% training accuracy (escaped plateau).
  • Case 4 (Avoiding Singularity via Init): 8,000 iterations, 92.4% training accuracy.

Notably, escaping a singularity (Case 3) required nearly double the training time (15,000 vs. 8,000 iterations) to reach optimal performance. The authors found that zero weight singularities ($R_1$) are the dominant cause of plateaus in EEG tasks, while overlap singularities ($R_2$) have a negligible effect.

Conclusion

The research highlights that initialization significantly influences whether a network reaches high accuracy quickly or gets trapped in a plateau. Future work should focus on designing initialization strategies or optimization algorithms that steer networks away from singular regions, leading to more reliable emotion recognition systems.

Summary updated

Abstract

p>Human emotion recognition is an important issue in human–computer interactions, and electroencephalograph (EEG) has been widely applied to emotion recognition due to its high reliability. In recent years, methods based on deep learning technology have reached the state-of-the-art performance in EEG-based emotion recognition. However, there exist singularities in the parameter space of deep neural networks, which may dramatically slow down the training process. It is very worthy to investigate the specific influence of singularities when applying deep neural networks to EEG-based emotion recognition. In this paper, we mainly focus on this problem, and analyze the singular learning dynamics of deep multilayer perceptrons theoretically and numerically. The results can help us to design better algorithms to overcome the serious influence of singularities in deep neural networks for EEG-based emotion recognition.</p>
BibTeX
@article{guo2021singular,
  author = {Weili Guo and Guangyu Li and Jianfeng Lu and Jian Yang},
  title = {{Singular Learning of Deep Multilayer Perceptrons for EEG-Based Emotion Recognition}},
  journal = {Frontiers of Computer Science},
  year = {2021},
  doi = {10.3389/fcomp.2021.786964},
}
Figure