Speech Based Emotion Prediction: Can a Linear Model Work?

Anda Ouyang, Ting Dang, Vidhyasaharan Sethu, Eliathamby Ambikairajah

Abstract

While the field of speech emotion recognition has pivoted toward complex deep learning architectures like LSTMs, this paper demonstrates that a simple linear Autoregressive Exogenous (ARX) model can achieve comparable—and sometimes superior—performance. By using a linear framework, significant interpretability is gained, allowing for the analysis of temporal dynamics and spectral characteristics of emotion prediction that "black box" models obscure. We validate this approach across three standard datasets, showing that a well-tuned linear model can match state-of-the-art performance while remaining computationally efficient.

Introduction

In affective computing, the goal is to predict continuous emotional states, typically arousal and valence, from speech. Recent trends favor complex non-linear back-ends, specifically Long Short-Term Memory (LSTM) networks, due to their ability to capture temporal dependencies. However, these models suffer from poor interpretability, high computational costs, and risks of overfitting due to their parameter density. This research investigates whether the relationship between speech features and emotion is sufficiently linear to be captured by a well-tuned linear model, potentially offering a more transparent and efficient alternative.

Methodology

The proposed approach utilizes an Autoregressive Exogenous (ARX) model, which functions as a two-part linear engine. The "AutoRegressive" (AR) component predicts the current emotional state based on a weighted sum of past labels, while the "eXogenous" (X) component predicts the state based on current and historical speech features.

Architecture of the proposed ARX model, illustrating the integration of parallel FIR filters for input features and an all-pole filter for the autoregressive component.

Mathematically, the model treats the system as a set of parallel Finite Impulse Response (FIR) filters feeding into a single all-pole filter. Training is conducted using L1-regularized least-squares (Lasso) via the ADMM algorithm. This regularization is critical as it enforces sparsity, performing automatic feature selection and enhancing model interpretability by highlighting only the most relevant acoustic features.

Experimental Setup

The model was validated on three standard databases representing diverse emotional contexts:

  1. RECOLA: Spontaneous interactions (AVEC 2016 subset).
  2. SEWA: Spontaneous data (AVEC 2017 subset).
  3. USC CreativeIT: Acted dyadic interactions. Acoustic features were extracted using the 88-dimensional eGeMAPS set, a standard baseline in the affective computing community.

Results and Discussion

Performance was evaluated using the Concordance Correlation Coefficient (CCC), where higher values indicate better agreement between predicted and ground-truth emotional trajectories. The ARX model demonstrated remarkable competitiveness against state-of-the-art non-linear systems. On the RECOLA dataset, the ARX model achieved a CCC of 0.783 for arousal, significantly outperforming a standard LSTM (0.599) using identical features. In the SEWA dataset, ARX surpassed the state-of-the-art LSTM for arousal with a CCC of 0.540 compared to 0.527, a 1.3% relative improvement.

Optimization of model performance (CCC) for RECOLA valence across varying AR and X model orders, highlighting the stability of the linear approach.

The AR component effectively acts as a learned "smart smoother," replacing generic post-processing filters with a task-specific low-pass response. Analysis of optimal model orders (as seen in Figure 2) revealed that emotional dependencies typically span 0.5 to 2 seconds in the label space. Furthermore, the L1 sparsity induced by the Lasso training identified Pitch (F0) and MFCC-related features as the most consistent predictors across databases, with mean spectral flux emerging as a high-weight predictor for arousal. These findings suggest that a large fraction of emotion-specific information in speech is linear, and complex models may only capture marginal additional information at the cost of significantly higher complexity and reduced transparency.

Conclusion

The success of the linear ARX model challenges the necessity of "black box" architectures for continuous emotion prediction. While issues like data non-stationarity remain a challenge, the interpretability and efficiency of linear models provide a strong foundation for future research. Subsequent work will explore non-stationary stochastic process models and leverage frequency response analysis to further refine the mapping between acoustic properties and emotional shifts.

Summary updated

Abstract

Speech based continuous emotion prediction systems have predominantly been based on complex non-linear back-ends, with an increasing attention on long-short term memory recurrent neural networks. While this has led to accurate predictions, complex models may suffer from issues with interpretability, model selection and overfitting. In this paper, we demonstrate that a linear model can capture most of the relationship between speech features and emotion labels in the continuous arousal-valence space. Specifically, an autoregressive exogenous model (ARX) is shown to be an effective backend. This approach is validated on three commonly used databases, namely RECOLA, SEWA and USC CreativeIT, and shown to be comparable in terms of performance to state-of-the-art LSTM systems. More importantly, this approach allows for the use of well-established linear system theory to aid with model interpretability.
BibTeX
@inproceedings{ouyang2019speech,
  author = {Anda Ouyang and Ting Dang and Vidhyasaharan Sethu and Eliathamby Ambikairajah},
  title = {{Speech Based Emotion Prediction: Can a Linear Model Work?}},
  booktitle = {Interspeech},
  year = {2019},
  doi = {10.21437/interspeech.2019-3149},
}
Figure