Abstract
While the field of speech emotion recognition has pivoted toward complex deep learning architectures like LSTMs, this paper demonstrates that a simple linear Autoregressive Exogenous (ARX) model can achieve comparable—and sometimes superior—performance. By using a linear framework, significant interpretability is gained, allowing for the analysis of temporal dynamics and spectral characteristics of emotion prediction that "black box" models obscure. We validate this approach across three standard datasets, showing that a well-tuned linear model can match state-of-the-art performance while remaining computationally efficient.
Introduction
In affective computing, the goal is to predict continuous emotional states, typically arousal and valence, from speech. Recent trends favor complex non-linear back-ends, specifically Long Short-Term Memory (LSTM) networks, due to their ability to capture temporal dependencies. However, these models suffer from poor interpretability, high computational costs, and risks of overfitting due to their parameter density. This research investigates whether the relationship between speech features and emotion is sufficiently linear to be captured by a well-tuned linear model, potentially offering a more transparent and efficient alternative.
Methodology
The proposed approach utilizes an Autoregressive Exogenous (ARX) model, which functions as a two-part linear engine. The "AutoRegressive" (AR) component predicts the current emotional state based on a weighted sum of past labels, while the "eXogenous" (X) component predicts the state based on current and historical speech features.
Mathematically, the model treats the system as a set of parallel Finite Impulse Response (FIR) filters feeding into a single all-pole filter. Training is conducted using L1-regularized least-squares (Lasso) via the ADMM algorithm. This regularization is critical as it enforces sparsity, performing automatic feature selection and enhancing model interpretability by highlighting only the most relevant acoustic features.
Experimental Setup
The model was validated on three standard databases representing diverse emotional contexts:
- RECOLA: Spontaneous interactions (AVEC 2016 subset).
- SEWA: Spontaneous data (AVEC 2017 subset).
- USC CreativeIT: Acted dyadic interactions. Acoustic features were extracted using the 88-dimensional eGeMAPS set, a standard baseline in the affective computing community.
Results and Discussion
Performance was evaluated using the Concordance Correlation Coefficient (CCC), where higher values indicate better agreement between predicted and ground-truth emotional trajectories. The ARX model demonstrated remarkable competitiveness against state-of-the-art non-linear systems. On the RECOLA dataset, the ARX model achieved a CCC of 0.783 for arousal, significantly outperforming a standard LSTM (0.599) using identical features. In the SEWA dataset, ARX surpassed the state-of-the-art LSTM for arousal with a CCC of 0.540 compared to 0.527, a 1.3% relative improvement.
The AR component effectively acts as a learned "smart smoother," replacing generic post-processing filters with a task-specific low-pass response. Analysis of optimal model orders (as seen in Figure 2) revealed that emotional dependencies typically span 0.5 to 2 seconds in the label space. Furthermore, the L1 sparsity induced by the Lasso training identified Pitch (F0) and MFCC-related features as the most consistent predictors across databases, with mean spectral flux emerging as a high-weight predictor for arousal. These findings suggest that a large fraction of emotion-specific information in speech is linear, and complex models may only capture marginal additional information at the cost of significantly higher complexity and reduced transparency.
Conclusion
The success of the linear ARX model challenges the necessity of "black box" architectures for continuous emotion prediction. While issues like data non-stationarity remain a challenge, the interpretability and efficiency of linear models provide a strong foundation for future research. Subsequent work will explore non-stationary stochastic process models and leverage frequency response analysis to further refine the mapping between acoustic properties and emotional shifts.
Summary updated