Abstract
Detecting emotional nuances in spontaneous human interaction remains a significant challenge for machine learning. This study presents a multimodal system that predicts the "mood" of a storyteller by combining audio, text sentiment, and physiological data. By optimizing the architecture of a neural network to ingest different data types at specific layers—placing low-level sensor data in earlier layers and abstract linguistic features in later ones—the researchers achieved a 0.92 AUC in narrative-level classification. These results suggest that strategic network topology and multimodal integration are critical for real-time emotion inference in natural settings.
Introduction
While humans naturally interpret emotion through a "plurality of meanings"—distinguishing, for instance, between a sarcastic and a sincere "thanks"—machines often struggle with such subtleties. Most emotion-detection research relies on "acted" datasets or controlled stimuli. This paper addresses the gap in spontaneous, natural human interaction by developing a system to infer "latent emotive content" from storytelling. Such technology could serve as a "social coach" for individuals with social disabilities, providing real-time feedback during high-stakes interactions.
Methodology
The researchers integrated three data streams: para-linguistic audio cues (pitch, energy), physiological signals (heart rate, movement, skin temperature), and text sentiment scores. A primary innovation was Topology Optimization. Rather than feeding all features into the input layer, the team experimented with feature entry points. They discovered that introducing low-level sensor data (e.g., acceleration) in early layers and high-level abstract data (e.g., text sentiment) in later layers significantly improved performance.
Datasets and Feature Selection
The study utilized a novel multimodal dataset comprising 31 natural narratives (15 happy, 16 sad) from 10 subjects, totaling 67 minutes. This data was segmented into 804 five-second windows, manually annotated using a simplified Plutchik’s emotion wheel. To manage the high dimensionality of the data, Sequential Forward Selection (SFS) was employed to identify the 10 most predictive features from an initial pool of 610.
Results and Discussion
The optimized neural network achieved a narrative-level AUC of 0.92, significantly outperforming baseline models such as Support Vector Machines (SVM) and Linear Discriminant Analysis. At the segment level (5-second windows), the model reached 47.3% accuracy, representing a 17.9% relative improvement over the baseline and performing significantly above chance for a three-class problem (positive, negative, neutral).
Key findings indicate that the "where" of feature introduction matters: physiological signals were most effective in early layers, while linguistic sentiment performed better near the output. Furthermore, increased physical movement and specific cardiovascular patterns were strong indicators of negative narratives. Interestingly, background noise (MFCC 12) was selected as a key feature, likely capturing emotional silences.
Conclusion
This research demonstrates that multimodality is essential for capturing the full emotional picture of human narrative. While limited by a small sample size, the success of the optimized topology provides a roadmap for real-time social intervention systems. Future work will focus on scaling the dataset and developing non-intrusive methods for delivering emotional feedback to users.
Summary updated