Sample Efficient Deep Reinforcement Learning for Dialogue Systems with Large Action Spaces

Gellért Weisz, Pawel Budzianowski, Pei-Hao Su, Milica Gasic

Abstract: In spoken dialogue systems, we aim to deploy artificial intelligence to build automated dialogue agents that can converse with humans. A part of this effort is the policy optimisation task, which attempts to find a policy describing how to respond to humans, in the form of a function taking the current state of the dialogue and returning the response of the system. In this paper, we investigate deep reinforcement learning approaches to solve this problem. Particular attention is given to actor-critic methods, off-policy reinforcement learning with experience replay, and various methods aimed at reducing the bias and variance of estimators. When combined, these methods result in the previously proposed ACER algorithm that gave competitive results in gaming environments. These environments however are fully observable and have a relatively small action set so in this paper we examine the application of ACER to dialogue policy optimisation. We show that this method beats the current state-of-the-art in deep learning approaches for spoken dialogue systems. This not only leads to a more sample efficient algorithm that can train faster, but also allows us to apply the algorithm in more difficult environments than before. We thus experiment with learning in a very large action space, which has two orders of magnitude more actions than previously considered. We find that ACER trains significantly faster than the current state-of-the-art.

Introduction

Spoken Dialogue Systems (SDS), such as virtual assistants, face a significant challenge: training an AI to converse naturally when the number of possible responses is vast. Traditionally, this has been managed by using "summary actions"—hand-coded abstractions that simplify the decision-making process but limit flexibility. This paper explores deep reinforcement learning (RL) to bridge this gap, focusing on the Actor-Critic with Experience Replay (ACER) algorithm. The authors demonstrate that ACER can learn complex policies much faster than previous neural methods and can train directly on "master" action spaces exceeding 1,000 possible actions.

The Challenge of Large Action Spaces

In SDS, the "policy" determines the system's next move based on the dialogue state. Training this policy via RL is difficult due to data scarcity (human interactions are expensive) and the "curse of dimensionality" in action spaces. While summary actions simplify the task, they require significant manual engineering. Moving to a "master" action space—where the system chooses specific, detailed actions directly—increases the action space by orders of magnitude (e.g., from ~20 to over 1,000 actions), making traditional methods like Gaussian Processes (GP-SARSA) computationally prohibitive.

Methodology: ACER for Dialogue

The authors adapt the ACER framework to dialogue policy optimisation. ACER combines several advanced RL techniques to ensure stable, fast learning:

  • Experience Replay: The agent stores past conversations and "re-studies" them, making it much more sample efficient than algorithms that learn from a conversation once and discard it.
  • Retrace Algorithm: A safe way to perform "off-policy" learning (learning from data generated by an older version of the AI) that reduces variance and prevents the model from diverging.
  • Trust Region Policy Optimization (TRPO): This acts as a "safety rail," ensuring that the AI's policy doesn't change too drastically in a single update, which prevents the "catastrophic forgetting" common in deep RL.
The ACER algorithm adapted for dialogue policy optimization.

To handle the massive 1,035-action master space, a specialized neural network architecture was developed. Instead of a single flat output, the network predicts a "summary" policy and a "payload" policy (the specific details to inform about) separately, which are then combined to form the final action distribution.

Specialized neural network architecture for handling large master action spaces by decomposing the policy into summary and payload components.

Experimental Setup and Key Results

The system was evaluated in the Cambridge Restaurants (CamInfo) domain using both an agenda-based user simulator and real-world human evaluation via Amazon Mechanical Turk (AMT), involving approximately 900 dialogues.

Scaling and Efficiency

ACER successfully trained on a master action space with 1,035 actions, two orders of magnitude larger than what was previously standard. It matched the performance of state-of-the-art Gaussian Processes (GP) but was significantly faster. A training task that took 8.63 days with GP was completed in just 6.45 hours with ACER.

Closing the Semantic Gap

Traditionally, human-written code was needed to bridge the gap between high-level intent and low-level system actions. This research shows that with the right RL algorithm, the AI can learn this mapping itself from scratch. Surprisingly, in noisy environments (e.g., speech recognition errors), training on the large master action space actually outperformed the simplified summary space because the AI learned to be more versatile than a rigid, hand-coded mapping.

Human Validation

In real-world tests with humans, the ACER model trained on the master action space achieved the highest rewards and the shortest dialogues, averaging 5.98 turns compared to 6.61 for the GP baseline.

Conclusion and Future Work

This work demonstrates that advanced deep RL algorithms like ACER can effectively close the "semantic gap" in dialogue systems. By enabling efficient training directly on large action spaces, it removes the need for restrictive hand-coded abstractions. Future directions include combining this approach with supervised learning (pre-training on human-to-human transcripts) to further boost initial performance before reinforcement learning takes over.

Summary updated

Abstract

In spoken dialogue systems, we aim to deploy artificial intelligence to build automated dialogue agents that can converse with humans. A part of this effort is the policy optimisation task, which attempts to find a policy describing how to respond to humans, in the form of a function taking the current state of the dialogue and returning the response of the system. In this paper, we investigate deep reinforcement learning approaches to solve this problem. Particular attention is given to actor-critic methods, off-policy reinforcement learning with experience replay, and various methods aimed at reducing the bias and variance of estimators. When combined, these methods result in the previously proposed ACER algorithm that gave competitive results in gaming environments. These environments however are fully observable and have a relatively small action set so in this paper we examine the application of ACER to dialogue policy optimisation. We show that this method beats the current state-of-the-art in deep learning approaches for spoken dialogue systems. This not only leads to a more sample efficient algorithm that can train faster, but also allows us to apply the algorithm in more difficult environments than before. We thus experiment with learning in a very large action space, which has two orders of magnitude more actions than previously considered. We find that ACER trains significantly faster than the current state-of-the-art.
BibTeX
@article{weisz2018sample,
  author = {Gellért Weisz and Pawel Budzianowski and Pei-Hao Su and Milica Gasic},
  title = {{Sample Efficient Deep Reinforcement Learning for Dialogue Systems with Large Action Spaces}},
  journal = {IEEE ACM Trans. Audio Speech Lang. Process.},
  year = {2018},
  url = {https://openreview.net/forum?id=TpkX-jYcTr6},
}
Figure