Introduction
Traditional affective state modeling often focuses on symmetric interactions where emotions are exchanged bi-directionally. This research, presented at INTERSPEECH 2010, investigates a distinct category of interaction characterized by unilateral intent. In these scenarios, one participant (the evoker) intentionally attempts to influence the affective state of another (the experiencer). Examples include a narrator building suspense for an audience or a virtual agent attempting to soothe a user.
Unilateral Intent and Participant Roles
The study categorizes interactions into narrative and conversational settings, assigning specific roles to the participants involved. This framework allows for the development of "one-sided" models that require data from only one participant to predict emotional outcomes.
Such models are highly practical for real-world applications where recording both parties is either technically unfeasible or restricted by privacy concerns (e.g., monitoring only a customer service agent rather than the caller).
Methodology
The researchers developed two primary models to identify "intensity peaks"—moments of significant emotional arousal or deviation from a neutral state.
- The Evoker Model (Narrative): This model analyzes the narrator's speech to predict "narrative peaks" experienced by the viewer.
- The Experiencer Model (Conversational): This model analyzes the speech of a human interlocutor to detect emotional spikes occurring during interactions with an artificial agent.
Feature Engineering
The analysis utilized both acoustic and lexical features:
- Acoustic Features: Included pitch, intensity, and speech rate (syllables per segment).
- Lexical Features: Employed Whissell’s Dictionary of Affect to derive intensity scores based on word valence and arousal. Additionally, the frequency of functional pronouns (e.g., "I", "you", "we") was tracked as a proxy for social engagement.
Classification was performed using a 10-second sliding window with Naïve Bayes and J48 Decision Tree algorithms.
Experimental Datasets
The study utilized two distinct corpora:
- VideoCLEF 2009 (Beeldenstorm): A collection of Dutch documentaries where annotators identified the top three moments of peak emotional intensity to establish ground truth.
- SEMAINE Corpus: Recordings of human-agent interactions featuring "Sensitive Artificial Listeners" with distinct emotional personas (e.g., "Obadiah" for depressive, "Spike" for angry).
Results and Feature Divergence
The experiments revealed a significant divergence in feature efficacy between the two settings. In narrative settings, lexical features proved far more predictive than acoustic ones. The authors hypothesize that narrators employ diverse and sometimes contradictory acoustic strategies (e.g., whispering vs. shouting) to create tension, which introduces noise into acoustic models.
As shown in Table 2, the J48 classifier using lexical features achieved a Recall@3 of 0.41, significantly outperforming the random baseline of 0.09. Conversely, in conversational settings, acoustic features were more effective. In natural dialogue, lexical markers like pronouns are ubiquitous, making them less distinctive during emotional peaks, whereas physiological changes in the voice provide a clearer signal. Interestingly, combining feature sets (fusion) did not always improve results, as acoustic "noise" in storytelling occasionally degraded lexical performance.
Conclusion
This research demonstrates the viability of one-sided affective modeling in scenarios of unilateral intent. By focusing on either the stimulus (evoker) or the response (experiencer), systems can effectively track emotional arcs without requiring full-duplex recordings. The findings suggest that future systems should prioritize lexical analysis for narrative content and acoustic analysis for conversational interactions.
Summary updated