Policy Networks with Two-Stage Training for Dialogue Systems

Mehdi Fatemi, Layla El Asri, Hannes Schulz, Jing He, Kaheer Suleman

Abstract

In this paper, we propose to use deep policy networks which are trained with an advantage actor-critic method for statistically optimised dialogue systems. First, we show that, on summary state and action spaces, deep Reinforcement Learning (RL) outperforms Gaussian Processes methods. Summary state and action spaces lead to good performance but require pre-engineering effort, RL knowledge, and domain expertise. In order to remove the need to define such summary spaces, we show that deep RL can also be trained efficiently on the original state and action spaces. Dialogue systems based on partially observable Markov decision processes are known to require many dialogues to train, which makes them unappealing for practical deployment. We show that a deep RL method based on an actor-critic architecture can exploit a small amount of data very efficiently. Indeed, with only a few hundred dialogues collected with a handcrafted policy, the actor-critic deep learner is considerably bootstrapped from a combination of supervised and batch RL. In addition, convergence to an optimal policy is significantly sped up compared to other deep RL methods initialized on the data with batch RL. All experiments are performed on a restaurant domain derived from the Dialogue State Tracking Challenge 2 (DSTC2) dataset.

Introduction

Dialogue management serves as the decision-making core of AI assistants, determining the system's next action based on the inferred user state. Traditionally, this has been modeled as a Partially Observable Markov Decision Process (POMDP). However, POMDPs face significant challenges for practical deployment: they are notoriously data-hungry, requiring thousands of dialogues to train, and they often necessitate manual engineering of "summary spaces" and "restricted action sets" to remain tractable.

This research addresses these limitations by leveraging Deep Reinforcement Learning (RL) to handle high-dimensional state and action spaces directly. Furthermore, the authors introduce a "two-stage" training strategy to accelerate convergence and reduce the reliance on extensive interaction data.

Methodology

The study evaluates several Deep RL architectures against the standard Gaussian Process SARSA (GPSARSA) baseline. The architectures include:

  • Deep Q-Networks (DQN) and Double DQN (DDQN): Value-based methods estimating the utility of state-action pairs.
  • Deep Advantage Actor-Critic (DA2C): A policy-gradient method utilizing an "Actor" for action selection and a "Critic" for performance evaluation relative to a baseline.

The Two-Stage Training Approach

The primary innovation is the Teacher DA2C (TDA2C). To avoid the "cold start" problem where an agent initially performs poorly, the policy network is first trained via Supervised Learning on a small set of human-rated dialogues from the Dialogue State Tracking Challenge 2 (DSTC2) dataset. This bootstrapping phase teaches the model valid dialogue patterns. Subsequently, the model transitions to Reinforcement Learning to refine its strategy through simulated interactions.

Experimental Setup

Experiments were conducted in a restaurant-seeking domain using the DSTC2 corpus. An agenda-based user simulator provided the training environment. For supervised bootstrapping, 706 "optimal" dialogues were extracted from the corpus.

Results and Discussion

Performance on Summary and Original Spaces

The results demonstrate that Deep RL methods significantly outperform traditional Gaussian Process approaches. On simplified summary spaces, DQN converged twice as fast as GPSARSA. More importantly, Deep RL models maintained high performance when operating on the original, high-dimensional belief states, effectively eliminating the need for manual summary space engineering.

Comparison of RL algorithms and the GPSARSA baseline.

As shown in Figure 3, DDQN and DA2C exhibit superior stability and faster convergence compared to standard DQN. The reduction of overestimation bias in DDQN proves particularly beneficial for dialogue tasks.

Efficiency of Two-Stage Training

The two-stage training approach (TDA2C) proved highly effective. By initializing with supervised learning, the model avoids "notoriously bad" actions from the outset, reaching an optimal policy in approximately 3,000 dialogues. In contrast, models starting from scratch or using only batch RL required 5,000 to 6,000+ dialogues to reach similar performance levels.

Impact of supervised bootstrapping on training efficiency.

Figure 5 illustrates the dramatic "bootstrap" effect: models initialized with supervised data (SupExptBatchDA2C and SupFullBatchDA2C) start with significantly higher average rewards and lower dialogue lengths compared to those starting from scratch (DA2C).

Conclusion

This work demonstrates that Deep RL, particularly when combined with supervised bootstrapping, offers a robust and efficient path for dialogue management. By moving away from hand-engineered summary spaces and utilizing two-stage training, developers can build systems that learn faster and perform better with less manual intervention. Future research will explore the scalability of these methods to multi-domain environments and the potential for end-to-end learning directly from raw text or speech.

Summary updated

Abstract

In this paper, we propose to use deep policy networks which are trained with an advantage actor-critic method for statistically optimised dialogue systems. First, we show that, on summary state and action spaces, deep Reinforcement Learning (RL) outperforms Gaussian Processes methods. Summary state and action spaces lead to good performance but require pre-engineering effort, RL knowledge, and domain expertise. In order to remove the need to define such summary spaces, we show that deep RL can also be trained efficiently on the original state and action spaces. Dialogue systems based on partially observable Markov decision processes are known to require many dialogues to train, which makes them unappealing for practical deployment. We show that a deep RL method based on an actor-critic architecture can exploit a small amount of data very efficiently. Indeed, with only a few hundred dialogues collected with a handcrafted policy, the actor-critic deep learner is considerably bootstrapped from a combination of supervised and batch RL. In addition, convergence to an optimal policy is significantly sped up compared to other deep RL methods initialized on the data with batch RL. All experiments are performed on a restaurant domain derived from the Dialogue State Tracking Challenge 2 (DSTC2) dataset.
BibTeX
@inproceedings{fatemi2016policy,
  author = {Mehdi Fatemi and Layla El Asri and Hannes Schulz and Jing He and Kaheer Suleman},
  title = {{Policy Networks with Two-Stage Training for Dialogue Systems}},
  booktitle = {SIGDIAL Conference},
  year = {2016},
  url = {https://openreview.net/forum?id=uoTekLUfwGX},
}
Figure