Abstract
Recently, emotion detection in conversations has become a prominent research topic in Natural Language Processing. This paper focuses on emotion detection in multi-speaker conversations, addressing two primary challenges: context-sensitive dependence and speaker-sensitive dependence. We propose a conversational graph-based convolutional neural network (ConGCN) that represents each utterance and speaker as nodes in a large heterogeneous graph. Contextual relationships are modeled as edges between utterances, while speaker identity is modeled via edges between utterances and speaker nodes. This approach recasts emotion detection as a node classification problem. Experimental results on a multi-modal, multi-speaker corpus demonstrate the effectiveness of the proposed approach.
Introduction
Emotion detection in dialogue is significantly more complex than analyzing isolated text. In a multi-speaker environment, an individual's emotional state is influenced by two main factors: Context-Sensitive Dependence (how previous remarks influence the current state) and Speaker-Sensitive Dependence (how an individual's personality dictates their response).
While previous research has largely focused on two-party dialogues, real-world interactions often involve multiple participants. Existing sequence-based models, such as Recurrent Neural Networks (RNNs), often struggle to capture the distinct "emotional footprint" of multiple speakers across a large dataset. This paper introduces ConGCN to bridge this gap by leveraging the structural advantages of graph neural networks.
Methodology: The ConGCN Architecture
The core innovation of this work is the representation of an entire conversational corpus as a heterogeneous graph. This allows the model to capture non-linear dependencies that traditional sequence models might miss.
Graph Construction
- Nodes: The graph consists of Utterance nodes ($u$), representing the content of the speech, and Speaker nodes ($s$), representing the individuals involved. By treating speakers as global nodes that persist across different conversations, the model learns a "global" representation of a person's emotional tendencies (e.g., learning that a specific character is more prone to irritability).
- Edges: Relationships are defined by two types of undirected edges:
- Context Edges: Connect utterances within the same conversation, weighted by angular similarity to represent topical relevance.
- Speaker Edges: Connect an utterance node to its corresponding speaker node. These are weighted using inverse speaking frequency to prevent the model from overfitting to dominant characters and ensure it learns effectively from infrequent speakers.
Feature Extraction and Learning
Utterance features are extracted using a combination of CNNs and Bi-LSTMs for textual data (GloVe embeddings) and Bi-LSTMs for acoustic features (MFSCs). A two-layer GCN then performs message passing, allowing utterance representations to be refined by both their conversational context and the learned "personality" embedding of the speaker.
Experimental Setup
The model was evaluated on the MELD (Multi-modal EmotionLines Dataset), which contains over 13,000 utterances from 1,432 conversations from the TV show Friends. This dataset is particularly challenging as it involves 304 different speakers and provides multi-modal data (text, audio, and visual).
Results and Discussion
ConGCN demonstrated superior performance across all tested modalities compared to strong baselines like DialogueRNN, ICON, and CMN.
As shown in the results, ConGCN achieved a weighted F1-score of 59.4% in multi-modal performance, significantly outperforming the state-of-the-art DialogueRNN (57.0%). The gains were even more pronounced in the acoustic modality, where ConGCN reached 42.2% F1 compared to DialogueRNN’s 34.0%, suggesting that the graph structure is highly effective at capturing emotional prosody cues linked to specific speakers.
Ablation studies confirmed the importance of the graph's components: removing either the context edges or the speaker nodes resulted in a measurable drop in performance, validating the hypothesis that both context and speaker identity are crucial for accurate emotion detection in multi-party settings.
Conclusion
ConGCN effectively addresses the challenges of multi-speaker emotion detection by modeling context and speaker dependencies through a heterogeneous graph. Future work may explore semi-supervised learning to leverage unlabeled data and investigate the model's generalization to "zero-shot" speakers not seen during training. This graph-based approach holds promise for other conversational NLP tasks where speaker identity and context are vital.
Summary updated