Composite Task-Completion Dialogue System via Hierarchical Deep Reinforcement Learning

Baolin Peng, Xiujun Li, Lihong Li, Jianfeng Gao, Asli Celikyilmaz, Sungjin Lee, Kam-Fai Wong

Abstract

Building a dialogue agent to fulfill complex tasks, such as travel planning, is challenging because the agent has to learn to collectively complete multiple subtasks. For example, the agent needs to reserve a hotel and book a flight so that there leaves enough time for commute between arrival and hotel check-in. This paper addresses this challenge by formulating the task in the mathematical framework of options over Markov Decision Processes (MDPs), and proposing a hierarchical deep reinforcement learning approach to learning a dialogue manager that operates at different temporal scales. The dialogue manager consists of: (1) a top-level dialogue policy that selects among subtasks or options, (2) a low-level dialogue policy that selects primitive actions to complete the subtask given by the top-level policy, and (3) a global state tracker that helps ensure all cross-subtask constraints be satisfied. Experiments on a travel planning task with simulated and real users show that our approach leads to significant improvements over three baselines, two based on handcrafted rules and the other based on flat deep reinforcement learning.

Introduction

Most current dialogue assistants excel at simple, single-domain tasks like checking the weather. However, real-world planning is often composite. For instance, planning a vacation involves booking a flight, a hotel, and a rental car—all of which are interdependent. These tasks present three major challenges: reward sparsity, where success signals are only received at the end of long conversations; slot constraints, where subtasks have global dependencies; and incoherence, where standard agents often jitter between subtasks, leading to a poor user experience.

Hierarchical Reinforcement Learning Framework

The proposed framework utilizes the Options Framework within Reinforcement Learning to split the dialogue manager into a two-level hierarchy.

Hierarchical reinforcement learning framework with internal critic and two-level policy.
  • Top-Level Policy (Meta-Controller): Selects a subtask (option) based on the global state.
  • Low-Level Policy (Controller): Executes primitive actions to complete the chosen subtask.
  • Internal Critic: Provides intrinsic rewards upon subtask completion, addressing the sparse reward problem by giving the agent more frequent feedback.
  • Global State Tracker: This module maintains information across all subtasks to ensure that constraints (like matching dates between flights and hotels) are respected.

Both levels are trained using Deep Q-Networks (DQN) with experience replay and target networks to stabilize learning.

Experimental Setup

The researchers utilized the Frames dataset, a human-human dialogue corpus, adapted for composite settings with inter-subtask constraints. A sophisticated user simulator was developed to train the agents against three user types (A, B, and C) with varying degrees of constraint rigidity. Finally, human evaluations were conducted with 12 real users to assess naturalness and coherence.

Results and Discussion

The HDRL approach (HRL) was compared against Rule-based and Flat RL (DQN) agents. HRL achieved a success rate of ~62% on complex user types, nearly double the ~35% achieved by the Flat RL agent. Furthermore, HRL completed tasks in an average of ~43 turns, significantly more efficient than the Rule+ agent (~108 turns).

Human evaluation results comparing RL and HRL agents across different user types.

In real-world testing, humans gave the HRL agent significantly higher ratings for coherence and naturalness. The hierarchy allows the agent to focus exploration on relevant actions for the current subtask, making learning faster and more robust to "difficult" users who change their minds.

Conclusion

This work demonstrates that hierarchical reinforcement learning effectively manages the complexity of composite dialogue tasks. By decomposing the problem and utilizing intrinsic motivation, the agent achieves higher success rates and better user satisfaction. Future work includes exploring deeper hierarchies and automated "option discovery" to reduce the need for manual task decomposition.

Summary updated

Abstract

Building a dialogue agent to fulfill complex tasks, such as travel planning, is challenging because the agent has to learn to collectively complete multiple subtasks. For example, the agent needs to reserve a hotel and book a flight so that there leaves enough time for commute between arrival and hotel check-in. This paper addresses this challenge by formulating the task in the mathematical framework of options over Markov Decision Processes (MDPs), and proposing a hierarchical deep reinforcement learning approach to learning a dialogue manager that operates at different temporal scales. The dialogue manager consists of: (1) a top-level dialogue policy that selects among subtasks or options, (2) a low-level dialogue policy that selects primitive actions to complete the subtask given by the top-level policy, and (3) a global state tracker that helps ensure all cross-subtask constraints be satisfied. Experiments on a travel planning task with simulated and real users show that our approach leads to significant improvements over three baselines, two based on handcrafted rules and the other based on flat deep reinforcement learning.
BibTeX
@inproceedings{peng2017composite,
  author = {Baolin Peng and Xiujun Li and Lihong Li and Jianfeng Gao and Asli Celikyilmaz and Sungjin Lee and Kam-Fai Wong},
  title = {{Composite Task-Completion Dialogue System via Hierarchical Deep Reinforcement Learning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2017},
  doi = {10.18653/v1/d17-1237},
}
Figure