Sample Efficient Deep Reinforcement Learning for Dialogue Systems With Large Action Spaces

Gellért Weisz, Pawel Budzianowski, Pei-Hao Su, Milica Gasic

This paper, published in 2018 by researchers at the University of Cambridge and DeepMind, tackles a long-standing bottleneck in Spoken Dialogue Systems (SDS): how to train an AI to talk to humans efficiently when the number of possible things it can say is massive.

TL;DR

The authors adapt the ACER (Actor-Critic with Experience Replay) algorithm to spoken dialogue systems, demonstrating that it can learn complex policies much faster than previous neural methods. Crucially, they show that ACER can train directly on "master" action spaces (over 1,000 possible actions) without needing the hand-coded "summary" actions that have traditionally limited the flexibility of dialogue agents.

What Problem Does This Paper Address?

In a Spoken Dialogue System (like Alexa or Siri), the "policy" is the brain that decides what the system should say next based on the current conversation state. Training this brain using Reinforcement Learning (RL) is notoriously difficult for two reasons:

  1. Data Scarcity: Collecting data from real humans is slow and expensive. The system must be "sample efficient"—it needs to learn a lot from very few interactions.
  2. Large Action Spaces: To keep things simple, researchers usually use "summary actions" (e.g., a generic "inform" action). A separate hand-coded script then decides which specific information to give. Training directly on the "master" action space (e.g., "inform about the address and phone number of a Turkish restaurant") is much harder because the number of combinations explodes.

Previous state-of-the-art methods like Gaussian Processes (GP-SARSA) were sample efficient but became computationally "prohibitive" as the action space grew.

Methods

The authors implement ACER, a sophisticated RL framework that combines several techniques to ensure stable, fast learning:

  • Experience Replay: The agent stores past conversations and "re-studies" them, making it much more sample efficient than algorithms that learn from a conversation once and discard it.
  • Retrace Algorithm: A safe way to perform "off-policy" learning (learning from data generated by an older version of the AI) that reduces variance and prevents the model from diverging.
  • Trust Region Policy Optimization (TRPO): This acts as a "safety rail," ensuring that the AI's policy doesn't change too drastically in a single update, which prevents the "catastrophic forgetting" common in deep RL.
  • Specialized NN Architecture: To handle the 1,035-action master space, they designed a neural network that predicts a "summary" policy and a "payload" policy (the specific details to inform about) separately, then combines them.

Datasets

  • Cambridge Restaurants (CamInfo): A domain where the system helps users find restaurants based on criteria like food type, area, and price range.
  • User Simulator: An agenda-based simulator used for initial training and large-scale benchmarking.
  • Human Evaluation: The final models were tested against real humans recruited via Amazon Mechanical Turk (AMT), involving approximately 900 dialogues.

Key Insights and Ideas

  • The "Semantic Gap" is Closed: Traditionally, we needed human-written code to bridge the gap between high-level intent and low-level system actions. This paper shows that with the right RL algorithm, the AI can learn this mapping itself from scratch.
  • Master Space > Summary Space: Surprisingly, when there is "noise" (like a crackly phone line or speech recognition errors), training on the large master action space actually outperforms the simplified summary space. This is because the AI learns to be more versatile than a rigid, hand-coded mapping.
  • Computational Efficiency: While Gaussian Processes (GP) are very sample efficient, they are computationally heavy. The authors found that a task taking 6.45 hours to train with ACER took 8.63 days with GP.

Key Results

  • Scaling Up: ACER successfully trained on a master action space with 1,035 actions, two orders of magnitude larger than what was previously standard.
  • Speed and Success: In the restaurant domain, ACER reached a ~97% success rate, matching the performance of Gaussian Processes but with significantly lower computational overhead.
  • Human Validation: In real-world tests with humans, the ACER model trained on the master action space achieved the highest rewards and the shortest dialogues (averaging 5.98 turns vs. 6.61 for the GP baseline).

Limitations and Future Work

  • Static Action Spaces: The current model assumes the list of possible actions (the "ontology") is fixed. If a new restaurant attribute is added to the database, the model would likely need to be retrained.
  • Modular vs. End-to-End: This work focuses on the "Dialogue Manager" module. While the authors note ACER is applicable to end-to-end models (where the AI goes directly from audio/text to response), they did not test that full pipeline here.
  • Future Direction: The authors suggest combining this approach with supervised learning (pre-training on human-to-human transcripts) to further boost initial performance before the RL takes over.

Summary updated

Abstract

In spoken dialogue systems, we aim to deploy artificial intelligence to build automated dialogue agents that can converse with humans. A part of this effort is the policy optimization task, which attempts to find a policy describing how to respond to humans, in the form of a function taking the current state of the dialogue and returning the response of the system. In this paper, we investigate deep reinforcement learning approaches to solve this problem. Particular attention is given to actor-critic methods, off-policy reinforcement learning with experience replay, and various methods aimed at reducing the bias and variance of estimators. When combined, these methods result in the previously proposed ACER algorithm that gave competitive results in gaming environments. These environments, however, are fully observable and have a relatively small action set so, in this paper, we examine the application of ACER to dialogue policy optimization. We show that this method beats the current state of the art in deep learning approaches for spoken dialogue systems. This not only leads to a more sample efficient algorithm that can train faster, but also allows us to apply the algorithm in more difficult environments than before. We thus experiment with learning in a very large action space, which has two orders of magnitude more actions than previously considered. We find that ACER trains significantly faster than the current state of the art.
BibTeX
@article{weisz2018sample,
  author = {Gellért Weisz and Pawel Budzianowski and Pei-Hao Su and Milica Gasic},
  title = {{Sample Efficient Deep Reinforcement Learning for Dialogue Systems With Large Action Spaces}},
  journal = {IEEE ACM Trans. Audio Speech Lang. Process.},
  year = {2018},
  url = {https://openreview.net/forum?id=TpkX-jYcTr6},
}

Loading related content...

Figure