TL;DR
This paper introduces ScenarioSA, a novel, large-scale dataset of 2,214 multi-turn English conversations, specifically designed to advance interactive sentiment analysis. By capturing the dynamic interplay and evolving sentiments between two speakers, ScenarioSA addresses a critical gap in existing datasets, which often overlook conversational context. Experiments on this dataset reveal that current sentiment analysis models struggle with interactive dynamics, highlighting the need for new approaches.
What Problem Does This Paper Address?
Traditional sentiment analysis typically focuses on isolated texts, like reviews or tweets, to determine sentiment. However, human communication is rarely so simple. In conversations, people influence each other, and their sentiments can shift dynamically based on the interaction. This is the domain of interactive sentiment analysis, which aims to understand not just individual sentiment but also how it evolves within a dialogue. The major hurdle in this field has been the lack of suitable datasets that capture these complex conversational dynamics. Existing datasets are often too small, lack detailed interaction information, or are not publicly available. ScenarioSA aims to fill this void by providing a rich, annotated dataset for researchers to develop and evaluate models that can truly understand sentiment in conversation.
Methods
The core contribution of this paper is the creation and release of the ScenarioSA dataset. The authors collected over 3,000 multi-turn English conversations from various online sources. These conversations were then meticulously pre-processed, including anonymizing speaker names and filtering out dialogues with more than two participants.
The crucial step was the manual annotation process. Annotators were tasked with two levels of labeling:
- Utterance-level sentiment: Each individual turn in the conversation was labeled as positive (1), negative (-1), or neutral (0).
- Conversation-level sentiment: The final affective state of each speaker at the end of the conversation was also labeled.
To ensure annotation quality, four volunteers with sentiment analysis expertise were recruited. They underwent a rigorous training phase, and inter-annotator agreement was measured, showing reasonable reliability.
The authors also conducted experiments using various state-of-the-art sentiment analysis models (CNN, LSTM, AT-LSTM, IAN) on ScenarioSA. They explored two strategies for aggregating utterance-level sentiments to derive a speaker's overall sentiment: simple summation and a learned weighting strategy. A novel model, IAN-INF, was also proposed, which incorporates learned "influence scores" into an LSTM's output gate to better model inter-speaker dynamics.
Datasets
The primary dataset introduced in this paper is ScenarioSA.
- Size: 2,214 multi-turn English conversations.
- Content: Over 24,000 utterances, covering a wide range of natural contexts (shopping, work, daily life).
- Annotation: Each utterance is labeled with sentiment polarity (positive, negative, neutral), and each speaker's final affective state is also annotated.
- Key Differentiators: Unlike many existing sentiment datasets, ScenarioSA explicitly models the interaction between two speakers and tracks their sentimental evolution throughout the conversation.
The authors also reference existing datasets like Movie Reviews and SemEval-2014 for comparative evaluation of their models, noting that their models perform significantly worse on ScenarioSA than on these more traditional datasets, underscoring the difficulty of interactive sentiment analysis.
Key Insights and Ideas
- The "Interaction Effect" is Real and Significant: The dataset analysis shows that a substantial portion of conversations (70.3%) exhibit a change in speakers' affective states due to interaction, proving that conversational context is not just noise but a crucial factor in sentiment.
- Current Models Struggle with Dynamics: State-of-the-art models, even those incorporating attention mechanisms, perform considerably worse on ScenarioSA compared to their performance on isolated text datasets. This strongly suggests that existing architectures are not well-equipped to handle the nuances of interactive sentiment.
- Modeling Social Influence is Key: The proposed IAN-INF model, which explicitly tries to capture influence between speakers, shows improved performance, indicating that explicitly modeling social dynamics is a promising direction for interactive sentiment analysis.
- Final Turn Isn't Enough: Relying solely on the sentiment of the last utterance to determine a speaker's overall sentiment is insufficient, especially when sentiments evolve significantly throughout the conversation.
Key Results
- Dataset Statistics: ScenarioSA contains 2,214 conversations, 24,072 utterances, and 228,047 words, with an average of 5.9 turns and 103 words per conversation.
- Model Performance: On ScenarioSA, models like IAN achieved accuracies of around 64.4% for speaker A and 70.7% for speaker B. This is a significant drop compared to their performance on SemEval-2014 (78.6% and 83.1%).
- IAN-INF Improvement: The IAN-INF model, which incorporates learned influence scores, achieved the best results among the tested models, reaching 65.8% accuracy for speaker A and 72.1% for speaker B (using weighted combination). While an improvement, these results still highlight the challenge.
- Annotation Agreement: The average agreement among annotators was about 73.2% at the utterance level and 78.6% at the conversation level, indicating a reasonable level of annotation quality.
Limitations and Future Work
- Model Limitations: The authors acknowledge that even their improved IAN-INF model does not perfectly capture the sentimental change of speakers, as illustrated by a qualitative example. This points to the need for entirely new methodologies.
- Dataset Scope: While ScenarioSA is large and diverse, it is limited to English conversations between two speakers. Future work could expand to multi-party conversations and other languages.
- Annotation Granularity: The sentiment labels are coarse-grained (positive, negative, neutral). More nuanced emotion detection could be explored.
- Future Directions: The authors emphasize the need for novel interactive sentiment analysis models that can better capture the dynamic influence and evolving sentiments within conversations. They suggest that ScenarioSA is a valuable resource for developing and testing such future models.
Summary updated