Introduction
Training deep neural networks for EEG-based emotion recognition often encounters frustrating "plateaus" where progress nearly stops. This research identifies specific mathematical "singularities" in the network's parameter space—specifically zero weights and overlapping neurons—as the primary cause of these delays. Using the benchmark SEED dataset, the authors demonstrate how these geometric properties of the loss landscape impact training efficiency and model performance.
Problem Statement: The Plateau Phenomenon
Deep learning is a standard tool for decoding human emotions from electroencephalograph (EEG) signals. However, researchers frequently face the "plateau phenomenon," where the loss function stops decreasing for extended periods before a sudden drop. This is not merely a hardware limitation but a fundamental issue with the "geometry" of the neural network. In regions called singularities, the Fisher information matrix becomes singular, causing standard gradient descent algorithms to lose direction and stall. This paper bridges the gap between theoretical singular learning theory and practical EEG applications.
Methods and Architecture
The study focuses on Deep Multilayer Perceptrons (MLPs) as a representative model for complex architectures.
Theoretical Analysis
The authors define two primary types of singularities in a network with two hidden layers:
- Zero Weight Singularity ($R_1$): Occurs when a weight connecting a hidden unit to the output layer becomes zero, effectively removing that path from the network.
- Overlap Singularity ($R_2$): Occurs when two hidden units have identical weights, making them redundant.
Dataset and Preprocessing
The SEED dataset (SJTU Emotion EEG Dataset) was used, featuring EEG signals from 15 subjects watching clips designed to elicit positive, neutral, or negative emotions. The 62-channel data was divided into five frequency bands. Principal Component Analysis (PCA) reduced the input to a 25-dimension vector for a 25-8-8-1 MLP architecture.
Key Insights and Results
Numerical simulations tracked the "learning trajectories" of weights during training to observe interactions with singular regions.
Plateaus as Saddle Points
The results indicate that plateaus caused by zero weight singularities are saddle points rather than local minima. The network eventually escapes, but the path is narrow and time-consuming.
Impact of Singularities
The study compared four cases to quantify the impact:
- Case 1 (Fast Convergence): 8,000 iterations, 94.8% training accuracy.
- Case 2 (Zero Weight Singularity): 8,000 iterations, 90.1% training accuracy (trapped in plateau).
- Case 3 (Extended Training): 15,000 iterations, 94.4% training accuracy (escaped plateau).
- Case 4 (Avoiding Singularity via Init): 8,000 iterations, 92.4% training accuracy.
Notably, escaping a singularity (Case 3) required nearly double the training time (15,000 vs. 8,000 iterations) to reach optimal performance. The authors found that zero weight singularities ($R_1$) are the dominant cause of plateaus in EEG tasks, while overlap singularities ($R_2$) have a negligible effect.
Conclusion
The research highlights that initialization significantly influences whether a network reaches high accuracy quickly or gets trapped in a plateau. Future work should focus on designing initialization strategies or optimization algorithms that steer networks away from singular regions, leading to more reliable emotion recognition systems.
Summary updated