Abstract
The performance of a task-completion dialogue agent usually affects the user experience: when the conversation system yields an unreasonable response, users may feel dissatisfied. Besides, early termination often occurs in disappointing conversations. However, existing off-the-shelf user simulators generally assume an ideal and cooperative user, which is somewhat different from a real user, and inevitably lead to a sub-optimal dialogue policy. In this paper, we propose an emotion-aware user simulation framework for task-oriented dialogue, which is based on the OCC emotion model to update user emotions and drive user actions, to generate simulated behaviors that more similar to real users. We present a linear implementation that is easy to understand and extend, and evaluate it on two domain-specific datasets. The experimental results show that the emotional simulation results of our proposed framework conform to common sense and have good versatility for different domains. Meanwhile, our framework provides us with another perspective to understand the improvement process of the dialogue policy model based on reinforcement learning.
Introduction
In task-oriented dialogue systems, such as those for booking taxis or movie tickets, the dialogue policy determines the system's response strategy. Training these policies typically relies on Reinforcement Learning (RL), which requires thousands of interactions. Due to the high cost of human-in-the-loop training, researchers rely on user simulators. However, most existing simulators assume an "ideal" user—one who is infinitely patient and perfectly cooperative. This creates a "reality gap": agents trained on such simulators fail to handle real-world users who may become frustrated by repetitive or irrelevant system queries and terminate the session prematurely. This paper introduces an emotion-aware simulator based on the OCC psychological model that experiences frustration and can prematurely terminate conversations, forcing RL agents to develop more efficient and user-friendly policies.
Methodology
The proposed framework mimics human behavior on two levels: task-completion and emotional state. The core of the approach is the OCC (Ortony, Clore, and Collins) emotion model, implemented through three procedures:
- Trigger Detection: The system monitors for "triggers" such as excessive dialogue length, irrelevant responses, or repeated queries.
- Emotional State Updating: The user’s emotional state is updated based on these triggers, modulated by the user's Big Five personality traits and a temporal decay factor.
- Action Driving: If negative emotional intensity (e.g., Anger, Disgust) crosses a threshold, the simulator has a probability of sending a "terminating" signal—effectively hanging up on the bot.
The authors provide a linear implementation of this model, using weighted matrices to map how specific events affect specific emotions based on personality.
Experimental Setup
The framework was evaluated on two domain-specific datasets: Movie-ticket booking (2,890 labeled conversations) and Taxi-ordering (3,094 labeled conversations). These datasets include standard intents (e.g., inform, request) and slots (e.g., moviename, pickup_location). The authors extended these by adding a "terminating" intent to represent early suspension of the conversation.
Results and Analysis
Experimental results demonstrate that emotional modeling introduces significant realism. Different personality profiles ($u_A$ and $u_B$) lead to distinct emotional trajectories even under identical dialogue conditions. For example, User A and User B showed distinct intensities of "Surprise" and "Happiness" despite the same task outcome.
Crucially, the introduction of early termination ($u_A + terminate$) makes the task harder for the RL agent. As shown in the results, the success rate of the dialogue policy often drops when the simulator is allowed to quit (e.g., from 0.452 to 0.425 in early training epochs for the Movie domain). This confirms that emotional "impatience" provides a more realistic and challenging training environment.
As training progresses from Epoch 100 to 300, the RL agent learns to prioritize efficiency. The framework allows researchers to observe why a policy is improving: for instance, the proportion of "Repeated Queries" (RQ) decreases while "User Happiness" increases, providing a more granular view of the learning process than success rates alone.
Conclusion
By incorporating emotional triggers and the possibility of early termination, this framework addresses the "reality gap" in dialogue policy training. The results align with common sense, showing that as agents become more efficient, simulated user satisfaction rises. Future work will address the limitations of manual weight tuning and explore more complex, non-linear emotional models for multi-domain tasks.
Summary updated