Introduction
In the field of affective computing, training robust models for emotion recognition relies heavily on continuous human annotations, often referred to as "traces." These traces typically measure dimensions such as arousal (emotional intensity) and valence (pleasantness). However, human-generated ground truth is inherently noisy. Annotators exhibit varying reaction delays—the time taken to perceive an emotion and physically adjust a slider—and subjective biases. Crucially, these delays are not constant; they fluctuate throughout a recording, leading to temporal misalignment between the audio signal and the labels. This noise significantly hinders the performance of machine learning models tasked with real-time prediction.
Tandem RNN Architecture
To address these challenges, this paper proposes a novel tandem Recurrent Neural Network (RNN) architecture designed to dynamically align and "fix" human annotations. The system consists of two primary components trained jointly:
- The Correction Model (B-GRU): A Bidirectional Gated Recurrent Unit (B-GRU) processes raw human annotations to produce "corrected" versions. This model adjusts the timing and values of the labels while preserving the underlying emotional intent expressed by the annotators.
- The Prediction Model (Linear): This component attempts to predict the corrected annotations using objective acoustic features, specifically log Mel-filterbanks.
The core innovation lies in the joint learning objective. The models are optimized using a specialized loss function based on the Concordance Correlation Coefficient (CCC) and a novel Cross-CCC. This framework forces the B-GRU to generate corrections that are physically "predictable" from the audio data, effectively using the AI as a mediator to align human perception with physical reality.
Sinc-based Signal Smoothing
A significant technical hurdle in this alignment is the discrepancy between the high-frequency nature of audio features and the slow-moving trajectories of human emotional shifts. To bridge this gap, the authors implement a Sinc Layer—a trainable low-pass filter.
As shown in the signal analysis, the Sinc layer effectively removes high-frequency noise from the predictions, ensuring the output trajectories are smooth and natural rather than jittery. The researchers employed a curriculum learning strategy, starting with a very low-cutoff frequency and gradually increasing the detail allowed as training progressed, which proved vital for stability.
Experimental Evaluation and Results
The proposed method was evaluated on two major benchmarks: RECOLA (controlled interactions) and SEWA (challenging "in-the-wild" interactions). The results demonstrate massive performance gains over traditional baselines:
- RECOLA Arousal: Prediction accuracy (CCC) increased from a baseline of .190 to .720.
- RECOLA Valence: Performance improved from .074 to .538.
- Inter-Annotator Agreement: Interestingly, the AI-corrected labels showed higher Cronbach’s alpha scores, suggesting the system successfully extracted the common emotional signal hidden behind individual human delays.
While the system performed robustly across French, Chinese, and German cultures, it faced challenges with the Hungarian subset of SEWA, which had the lowest initial human consensus. This suggests that the model requires a baseline level of human agreement to function effectively.
Conclusion
By treating AI as a "super-annotator," this research provides a powerful method for generating high-quality Gold Standard data from messy human labels. The resulting lightweight linear models outperform complex state-of-the-art architectures simply by virtue of being trained on superior, time-aligned data. Future work will explore incorporating visual descriptors (facial expressions) and physiological signals to further refine the dynamic alignment process.
Summary updated