Neural Architecture Search for Speech Emotion Recognition

Xixin Wu, Shoukang Hu, Zhiyong Wu, Xunying Liu, Helen Meng

TL;DR

This paper introduces Neural Architecture Search (NAS) to the field of Speech Emotion Recognition (SER), moving away from manual, trial-and-error model design. The authors propose a "uniform path dropout" strategy for differentiable NAS that accelerates training and prevents the model from settling on sub-optimal architectures, ultimately achieving better accuracy with smaller model sizes.

What Problem Does This Paper Address?

Designing effective deep learning models for Speech Emotion Recognition (SER) is traditionally a "black art." Researchers usually rely on expert intuition and exhaustive manual tuning to decide on the number of layers, kernel sizes, and connection types. This process is not only time-consuming but often results in models that are larger and more computationally expensive than necessary.

While Neural Architecture Search (NAS) has automated design in computer vision, its application in SER has been limited. Existing differentiable NAS methods (like DARTS) often suffer from a "rich get richer" problem: the search process might prematurely favor simple, sub-optimal operations (like small kernels) before more complex, effective ones have had a chance to be properly trained.

Methods

The authors build upon Differentiable Architecture Search (DARTS). In this framework, a "supernet" is constructed containing all possible operations (e.g., different convolution kernel sizes) in parallel. The goal is to learn "architecture weights" that decide which operations are most effective.

The core novelty is the Uniform Path Dropout strategy:

  • The Standard Approach: Usually, NAS either optimizes everything at once (Joint Optimization) or randomly selects just one path to train at a time (Uniform Path Sampling).
  • The Proposed Approach: Instead of picking one path, the authors randomly "mask" or drop a constant fraction of operations (e.g., 1 out of 6) in a layer. They then optimize all the remaining paths simultaneously.
  • Why it works: By dropping a group of paths, the supernet is forced to stop relying on a single "favorite" sub-optimal operation. Because multiple paths are updated in every step, training is significantly faster than single-path sampling.

The authors applied this to two state-of-the-art SER architectures:

  1. CNN_RNN_att: A combination of CNNs, a Bi-GRU, and an attention mechanism.
  2. CNN_SeqCap: A CNN-based system utilizing Capsule Networks.

Datasets

The researchers evaluated their approach on the IEMOCAP (Interactive Emotional Dyadic Motion Capture) database.

  • Scope: They focused on four emotions: Neutral, Angry, Happy, and Sad.
  • Setup: They used five-fold cross-validation, using 8 speakers for training, 1 for validation (to optimize architecture weights), and 1 for testing.
  • Input: Speech signals were converted into spectrograms and split into 2-second segments.

Key Insights and Ideas

  • Breaking Dependency: Dropping paths acts as a regularizer for the architecture search. It prevents the supernet from becoming "lazy" and relying on operations that provide quick but mediocre gains early in training.
  • Efficiency vs. Stability: While "operation dropout" (dropping individual operations) exists, it can be unstable if a node accidentally drops all its operations. The authors' "layer-level" constant dropout ensures the network always has a path to learn through, maintaining training stability.
  • NAS as a Compressor: The search space included smaller kernel sizes than the baseline. The NAS was able to find configurations that used these smaller kernels effectively, proving that automated search can find "leaner" models that humans might overlook.

Key Results

The proposed NAS method outperformed both manual baselines and standard random search:

  • Accuracy Boost: For the CNN_RNN_att system, Unweighted Accuracy (UA) improved from 54.89% to 56.28%.
  • Model Efficiency: In the CNN_SeqCap system, the NAS-found model achieved a higher Weighted Accuracy (70.54% vs 69.86%) while actually reducing the parameter count (from 704k to 700k).
  • Superiority of Dropout: The "Uniform Path Dropout" consistently yielded lower loss and higher accuracy compared to "Uniform Path Sampling," confirming that optimizing multiple paths simultaneously is more effective for SER tasks.

Limitations and Future Work

The authors acknowledge a few areas for growth:

  • Search Space Diversity: The current search was limited to specific parameters like kernel sizes and dense layer dimensions. Future work could explore more diverse operations, such as different activation functions or non-linear connections.
  • Transferability: The paper focuses on the IEMOCAP dataset. It remains to be seen how well an architecture "discovered" on one dataset transfers to other speech corpora or real-world, noisy environments.
  • Hardware-Awareness: While they reduced parameter counts, they did not explicitly include hardware metrics (like latency or energy consumption) into the search objective.

Summary updated

Abstract

Deep neural networks have brought significant advancements to speech emotion recognition (SER). However, the architecture design in SER is mainly based on expert knowledge and empirical (trial-and-error) evaluations, which is time-consuming and resource intensive. In this paper, we propose to apply neural architecture search (NAS) techniques to automatically configure the SER models. To accelerate the candidate architecture optimization, we propose a uniform path dropout strategy to encourage all candidate architecture operations to be equally optimized. Experimental results of two different neural structures on IEMOCAP show that NAS can improve SER performance (54.89% to 56.28%) while maintaining model parameter sizes. The proposed dropout strategy also shows superiority over the previous approaches.
BibTeX
@inproceedings{wu2022neural,
  author = {Xixin Wu and Shoukang Hu and Zhiyong Wu and Xunying Liu and Helen Meng},
  title = {{Neural Architecture Search for Speech Emotion Recognition}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=wpBssSnhoDu},
}

Loading related content...

Figure