TL;DR
This paper introduces a novel multimodal dialogue system designed to foster consensus-building between humans and AI. By integrating a slot-filling strategy for structured discussion and an avatar that displays emotions and intentions, the system successfully guided negotiations towards agreement, outperforming other systems in a competition.
What Problem Does This Paper Address?
The increasing sophistication of large language models (LLMs) has enabled more fluent human-AI conversations. However, many real-world applications require dialogue systems to go beyond simple information exchange and actively influence user behavior through negotiation, persuasion, and consensus-building. This is particularly challenging when the AI and human have conflicting goals. Existing research has limited exploration of how to effectively impact users' cognitive and emotional responses to achieve consensus. The Dialogue System Live Competition 6 (DSLC6) was organized to address this gap, evaluating systems on their ability to build consensus in a scenario where an AI and a user jointly plan a party with differing preferences (grand vs. modest).
Methods
The proposed system comprises two main modules: a response-generation module and an avatar-control module.
- Response-Generation Module: This module leverages GPT-4 to generate system utterances and corresponding emotion/pose labels.
- Prompt Engineering: The system prompt is carefully crafted to include the dialogue's basic situation, agenda, intended objective, example utterances for output formatting, and the current state of predefined subtopic slots. A key modification was adding instructions to encourage empathy and discussion when opinions differ, preventing the system from readily accepting conflicting user proposals.
- Slot-Filling Dialogue State Management: To guide discussions, the system employs a slot-filling strategy. GPT-4 is used individually for each subtopic to dynamically update slots based on the dialogue history. This allows the system to initiate discussions on unfilled subtopics and transition smoothly to the next once a conclusion is reached.
- Emotion and Pose Label Generation: Unlike systems that separate utterance generation from emotion classification, this system simultaneously generates both. This ensures coherence between the spoken content and the avatar's emotional expression. Candidate emotion labels are derived from Plutchik's eight basic emotions plus "Neutral." Four pose labels (Bowing, Nodding, Shaking head, Pondering) are used to convey intent and prevent misunderstandings.
- Avatar-Control Module: This module translates the generated emotion/pose labels and utterance content into multimodal outputs for the avatar.
- Voice Modulation: Parameters like speed, volume, and pitch are adjusted based on emotion labels, mapped to Russell's Circumplex Model of Affect. Pauses are also strategically introduced to convey emotion and allow for nuance.
- Facial Expressions: A predefined set of facial expressions corresponding to the nine emotion labels is used.
- Pose Control: Poses are determined by both labels and utterance content, incorporating fundamental behaviors like nodding and specific gestures for emotions like "Pondering" (hand-on-chin) and "Surprise" (hand over mouth).
Datasets
The system utilizes resources provided by the Dialogue System Live Competition 6 (DSLC6). The emotion labels are based on Plutchik's basic emotions, with the Japanese WRIME dataset (Kajiwara et al., 2021) being a reference for emotion intensity estimation. The system also uses AmazonPollyServer for its Text-to-Speech (TTS) function.
Key Insights and Ideas
- Simultaneous Emotion and Utterance Generation: Generating emotion and pose labels alongside the utterance is crucial for ensuring multimodal coherence and avoiding inconsistencies that can arise from separate classification.
- Avatar as an Emotional Communicator: Effectively conveying emotions through an avatar's voice, facial expressions, and poses significantly influences user behavior and is vital for trust development and consensus building in negotiations.
- Structured Dialogue Guidance: A slot-filling approach, where the system breaks down a large topic into subtopics and guides the user through them, is an effective strategy for leading discussions and managing consensus-building processes.
- Prompt Engineering for Empathy: Explicitly instructing the LLM to engage empathetically and discuss differing opinions is key to preventing it from passively accepting user proposals that conflict with its goals.
- Streaming for Responsiveness: Employing a streaming mode for response generation significantly reduces perceived latency, making the system feel more responsive and less likely to interrupt the user.
Key Results
The system achieved the highest performance in the DSLC6 competition, outperforming other participating systems across all evaluation criteria: content, expressions, and voice modulation. Evaluators noted that the system effectively conveyed its intentions and progressed the dialogue well, with conversations and gestures feeling "very human." However, some feedback indicated that the system spoke "too quickly" and its expressions could sometimes be "exaggerated."
Limitations and Future Work
The authors acknowledge that while the system performed well, there is still room for improvement, particularly regarding the perceived speed of speech and the intensity of non-verbal expressions. Future work could focus on refining these aspects to create an even more natural and less fatiguing interaction experience. The modular design, with task-specific prompts and slots, suggests that the system is adaptable to various consensus-building scenarios with minimal re-engineering effort.
Summary updated