Neural Architecture Search for Speech Emotion Recognition

Xixin Wu, Shoukang Hu, Zhiyong Wu, Xunying Liu, Helen Meng

Abstract: Deep neural networks have brought significant advancements to speech emotion recognition (SER). However, the architecture design in SER is mainly based on expert knowledge and empirical (trial-and-error) evaluations, which is time-consuming and resource intensive. In this paper, we propose to apply neural architecture search (NAS) techniques to automatically configure the SER models. To accelerate the candidate architecture optimization, we propose a uniform path dropout strategy to encourage all candidate architecture operations to be equally optimized. Experimental results of two different neural structures on IEMOCAP show that NAS can improve SER performance (54.89% to 56.28%) while maintaining model parameter sizes. The proposed dropout strategy also shows superiority over the previous approaches.

Introduction

Designing effective deep learning models for Speech Emotion Recognition (SER) has traditionally been a manual, resource-intensive process. Researchers often rely on expert intuition and exhaustive trial-and-error to determine optimal hyperparameters, such as the number of layers, kernel sizes, and connection types. This "black art" of manual tuning frequently leads to oversized models that are computationally expensive and potentially sub-optimal.

While Neural Architecture Search (NAS) has successfully automated design in fields like computer vision, its application in SER remains limited. A significant challenge in existing differentiable NAS methods, such as DARTS, is the "rich get richer" problem: the search process may prematurely favor simple operations (like small kernels) before more complex, potentially more effective operations have been sufficiently trained. This paper addresses these challenges by introducing a novel dropout strategy to ensure balanced optimization across the search space.

Methodology

The proposed approach builds upon Differentiable Architecture Search (DARTS). In this framework, a "supernet" is constructed containing all possible operations in parallel for each layer. The goal is to learn "architecture weights" that determine the most effective operations for the final model.

The structure of a searchable layer in the supernet, where architecture weights (a) determine the contribution of different operations.

Uniform Path Dropout

The core novelty of this work is the Uniform Path Dropout strategy. Traditional NAS methods typically use either joint optimization (training all paths) or uniform path sampling (training one random path at a time). The authors argue that these can lead to instability or biased optimization.

Instead, the proposed strategy randomly "masks" or drops a constant fraction of operations (e.g., 1 out of 6) in a layer and optimizes all remaining paths simultaneously. This approach offers two primary benefits:

  1. Regularization: By dropping groups of paths, the supernet is forced to stop relying on a single "favorite" sub-optimal operation early in training.
  2. Efficiency: Because multiple paths are updated in every step, training is significantly faster than single-path sampling while remaining more stable than dropping individual operations.
Comparison between (a) Uniform Path Sampling, which selects a single path, and (b) the proposed Uniform Path Dropout, which optimizes multiple paths simultaneously.

Experimental Setup and Results

The researchers evaluated their approach using the IEMOCAP (Interactive Emotional Dyadic Motion Capture) database, focusing on four emotions: Neutral, Angry, Happy, and Sad. They applied NAS to two state-of-the-art architectures: CNN_RNN_att (CNNs with Bi-GRU and attention) and CNN_SeqCap (CNNs with Capsule Networks).

Key Findings

  • Performance Gains: For the CNN_RNN_att system, Unweighted Accuracy (UA) improved from 54.89% to 56.28%.
  • Model Efficiency: In the CNN_SeqCap system, the NAS-discovered model achieved higher Weighted Accuracy (70.54% vs 69.86%) while slightly reducing the parameter count (from 704k to 700k). This demonstrates that NAS can find "leaner" configurations, such as specific kernel sizes, that human designers might overlook.
  • Optimization Stability: The Uniform Path Dropout strategy consistently yielded lower loss and higher accuracy compared to standard sampling, confirming that simultaneous multi-path optimization is more effective for the SER task.

Conclusion

This research demonstrates that NAS can effectively automate and improve the design of SER models, moving beyond manual heuristics. The proposed Uniform Path Dropout strategy acts as a powerful regularizer, preventing premature convergence to sub-optimal architectures. Future work will explore expanding the search space to include diverse activation functions and non-linear connections, as well as incorporating hardware-aware metrics like latency into the search objective.

Summary updated

Abstract

Deep neural networks have brought significant advancements to speech emotion recognition (SER). However, the architecture design in SER is mainly based on expert knowledge and empirical (trial-and-error) evaluations, which is time-consuming and resource intensive. In this paper, we propose to apply neural architecture search (NAS) techniques to automatically configure the SER models. To accelerate the candidate architecture optimization, we propose a uniform path dropout strategy to encourage all candidate architecture operations to be equally optimized. Experimental results of two different neural structures on IEMOCAP show that NAS can improve SER performance (54.89\% to 56.28\%) while maintaining model parameter sizes. The proposed dropout strategy also shows superiority over the previous approaches.
BibTeX
@inproceedings{wu2022neural,
  author = {Xixin Wu and Shoukang Hu and Zhiyong Wu and Xunying Liu and Helen Meng},
  title = {{Neural Architecture Search for Speech Emotion Recognition}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=wpBssSnhoDu},
}
Figure