EmoNet: A Lightweight Attention-Based CNN for Speech Emotion Recognition

Muhammad Moiz Khalid, Abdur Rehman Tariq, Muhammad Okasha YaminarXiv 2026 · arXiv

TL;DR

EmoNet is a lightweight deep learning framework designed for Speech Emotion Recognition (SER) that achieves a high test accuracy of 96.75% with only 4.11 million parameters. By combining depthwise separable convolutions, squeeze-and-excitation attention, and temporal modeling, it offers a high-performance alternative to massive transformer models, making it ideal for real-time deployment on edge devices.

What Problem Does This Paper Address?

Speech Emotion Recognition (SER) is a vital component of affective computing, with applications ranging from mental health monitoring to automotive safety. However, the field faces a significant "efficiency gap." State-of-the-art models often rely on massive, pre-trained transformer architectures like Wav2Vec 2.0, which can contain hundreds of millions of parameters.

These "heavyweight" models are computationally expensive and difficult to deploy in real-time or on resource-constrained hardware (like smartphones or embedded systems in cars). Furthermore, many existing models struggle to generalize across different datasets due to variations in speaker styles and recording conditions. The authors of EmoNet set out to build a model that is both highly accurate and small enough for practical, real-world use.

Methods

The researchers developed a multi-stage pipeline that balances sophisticated feature extraction with an efficient neural architecture.

Fig. 1: Proposed pipeline for Speech Emotion Classification

1. Feature Extraction

Instead of using raw audio or simple MFCCs, the authors extracted a comprehensive 134-channel feature set. This includes 40 MFCCs along with their first and second-order derivatives (deltas and delta-deltas) to capture spectral dynamics, plus Chroma features for pitch, Zero Crossing Rate (ZCR), and Root Mean Square (RMS) energy.

2. The EmoNet Architecture

The core of the paper is the EmoNet architecture, which uses four distinct stages to process speech:

  • DS-CNN Encoder: A four-block encoder using Depthwise Separable Convolutions (DS-Conv). This technique factorizes standard convolutions into two smaller operations, drastically reducing the parameter count.
  • Squeeze-and-Excitation (SE) Attention: Integrated into the encoder blocks, SE modules allow the network to perform channel-wise attention, re-weighting informative spectral bands while suppressing noise.
  • Bidirectional LSTM (BiLSTM): Since emotion unfolds over time, a 2-layer BiLSTM is used to capture long-range temporal dependencies in both forward and backward directions.
  • Multi-Head Attention (MHA) Pooling: Rather than simply averaging the output over time, an 8-head attention mechanism "listens" for the most emotionally salient frames in an utterance.
Fig. 2: Proposed EmoNet Architecture

3. Training Strategy

The model was trained in two phases (Main and Fine-tuning) using a suite of regularizers including MixUp (label interpolation) and SpecAugment (masking frequency/time blocks) to ensure the model doesn't overfit to specific recording environments.

Datasets

The authors evaluated EmoNet on a composite corpus of three popular datasets to ensure better generalization:

  • TESS: 2,800 samples from female speakers in a studio setting.
  • RAVDESS: 1,440 samples from professional actors (mixed gender).
  • SAVEE: 480 samples from male speakers with British accents.

This combined approach provides a diverse mix of genders, accents, and recording qualities.

Key Insights and Ideas

  • Efficiency through Factorization: The move from standard convolutions to Depthwise Separable Convolutions is the primary driver of EmoNet's small size. It allows the model to maintain a deep hierarchy without the "parameter explosion" seen in traditional CNNs.
  • Attention at Two Levels: EmoNet uses attention twice—once in the spectral domain (SE modules) to find the right frequencies, and once in the temporal domain (MHA pooling) to find the right moments in time.
  • Temporal Resolution Preservation: The authors intentionally omitted pooling in the final CNN block. This preserves the temporal resolution of the features before they reach the BiLSTM, providing a richer sequence for the model to analyze.
  • The Power of Augmentation: By using both audio-level perturbations (noise, pitch shifts) and feature-level masking (SpecAugment), the model becomes robust enough to handle the "domain shift" that occurs when moving between different datasets.

Key Results

EmoNet achieved impressive quantitative results, outperforming its own reproduced baseline while being significantly smaller:

  • Accuracy: 96.75% on the combined test set.
  • Parameter Count: Only 4.11 million parameters, which is approximately 34x fewer than the 139.24 million parameters in the baseline model.
  • Class Performance: All seven emotion classes (Angry, Disgust, Fear, Happy, Neutral, Sad, Surprise) achieved F1 scores between 0.95 and 0.99.
  • Cross-Domain Robustness: The model achieved 100% accuracy on TESS and remained strong on the more challenging SAVEE (90.79%) and RAVDESS (92.73%) subsets.
Fig. 3: Confusion Matrix

Limitations and Future Work

While EmoNet performs exceptionally well on acted datasets, the authors acknowledge that real-world speech is often naturalistic or spontaneous, which can differ significantly from studio recordings.

Future work will focus on:

  • Testing the model on more naturalistic corpora like CREMA-D and IEMOCAP.
  • Evaluating performance on spontaneous conversational speech.
  • Further refining the augmentation pipeline and exploring even more lightweight architectures for extreme edge deployment.

Summary updated

Abstract

Speech Emotion Recognition (SER) is a fundamental task in affective computing that enables applications such as mental health assessment, human-computer interaction, and automated customer support systems. Accurate emotion classification from speech remains challenging due to variations in speaker style, acoustic conditions, and the high computational cost of modern deep learning approaches. Many existing systems depend on large transformer-based architectures that require substantial computational resources, making them less suitable for real-time or resource-constrained environments. This research proposes EmoNet, a lightweight deep learning framework for speech emotion classification across seven emotion categories. The model is built on a four-block depthwise separable CNN encoder with squeeze-and-excitation attention and residual connections, combined with a Bidirectional LSTM for temporal feature learning and a multi-head attention pooling mechanism for improved representation. A comprehensive 134-channel acoustic feature set alongside data augmentation techniques including audio-level perturbations and SpecAugment is employed to improve robustness under diverse acoustic conditions. Evaluated on a combined dataset of TESS, RAVDESS, and SAVEE, EmoNet achieves a test accuracy of 96.75% with only 4.11 million parameters. The proposed model outperforms its reproduced baseline and remains competitive with significantly larger state-of-the-art architectures.
BibTeX
@misc{khalid2026emonet,
  author = {Muhammad Moiz Khalid and Abdur Rehman Tariq and Muhammad Okasha Yamin},
  title = {{EmoNet: A Lightweight Attention-Based CNN for Speech Emotion Recognition}},
  year = {2026},
  eprint = {2610.33977},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.33977},
}
Figure