Abstract: In spoken dialogue systems, we aim to deploy artificial intelligence to build automated dialogue agents that can converse with humans. A part of this effort is the policy optimisation task, which attempts to find a policy describing how to respond to humans, in the form of a function taking the current state of the dialogue and returning the response of the system. In this paper, we investigate deep reinforcement learning approaches to solve this problem. Particular attention is given to actor-critic methods, off-policy reinforcement learning with experience replay, and various methods aimed at reducing the bias and variance of estimators. When combined, these methods result in the previously proposed ACER algorithm that gave competitive results in gaming environments. These environments however are fully observable and have a relatively small action set so in this paper we examine the application of ACER to dialogue policy optimisation. We show that this method beats the current state-of-the-art in deep learning approaches for spoken dialogue systems. This not only leads to a more sample efficient algorithm that can train faster, but also allows us to apply the algorithm in more difficult environments than before. We thus experiment with learning in a very large action space, which has two orders of magnitude more actions than previously considered. We find that ACER trains significantly faster than the current state-of-the-art.
Introduction
Spoken Dialogue Systems (SDS), such as virtual assistants, face a significant challenge: training an AI to converse naturally when the number of possible responses is vast. Traditionally, this has been managed by using "summary actions"—hand-coded abstractions that simplify the decision-making process but limit flexibility. This paper explores deep reinforcement learning (RL) to bridge this gap, focusing on the Actor-Critic with Experience Replay (ACER) algorithm. The authors demonstrate that ACER can learn complex policies much faster than previous neural methods and can train directly on "master" action spaces exceeding 1,000 possible actions.
The Challenge of Large Action Spaces
In SDS, the "policy" determines the system's next move based on the dialogue state. Training this policy via RL is difficult due to data scarcity (human interactions are expensive) and the "curse of dimensionality" in action spaces. While summary actions simplify the task, they require significant manual engineering. Moving to a "master" action space—where the system chooses specific, detailed actions directly—increases the action space by orders of magnitude (e.g., from ~20 to over 1,000 actions), making traditional methods like Gaussian Processes (GP-SARSA) computationally prohibitive.
Methodology: ACER for Dialogue
The authors adapt the ACER framework to dialogue policy optimisation. ACER combines several advanced RL techniques to ensure stable, fast learning:
- Experience Replay: The agent stores past conversations and "re-studies" them, making it much more sample efficient than algorithms that learn from a conversation once and discard it.
- Retrace Algorithm: A safe way to perform "off-policy" learning (learning from data generated by an older version of the AI) that reduces variance and prevents the model from diverging.
- Trust Region Policy Optimization (TRPO): This acts as a "safety rail," ensuring that the AI's policy doesn't change too drastically in a single update, which prevents the "catastrophic forgetting" common in deep RL.
To handle the massive 1,035-action master space, a specialized neural network architecture was developed. Instead of a single flat output, the network predicts a "summary" policy and a "payload" policy (the specific details to inform about) separately, which are then combined to form the final action distribution.
Experimental Setup and Key Results
The system was evaluated in the Cambridge Restaurants (CamInfo) domain using both an agenda-based user simulator and real-world human evaluation via Amazon Mechanical Turk (AMT), involving approximately 900 dialogues.
Scaling and Efficiency
ACER successfully trained on a master action space with 1,035 actions, two orders of magnitude larger than what was previously standard. It matched the performance of state-of-the-art Gaussian Processes (GP) but was significantly faster. A training task that took 8.63 days with GP was completed in just 6.45 hours with ACER.
Closing the Semantic Gap
Traditionally, human-written code was needed to bridge the gap between high-level intent and low-level system actions. This research shows that with the right RL algorithm, the AI can learn this mapping itself from scratch. Surprisingly, in noisy environments (e.g., speech recognition errors), training on the large master action space actually outperformed the simplified summary space because the AI learned to be more versatile than a rigid, hand-coded mapping.
Human Validation
In real-world tests with humans, the ACER model trained on the master action space achieved the highest rewards and the shortest dialogues, averaging 5.98 turns compared to 6.61 for the GP baseline.
Conclusion and Future Work
This work demonstrates that advanced deep RL algorithms like ACER can effectively close the "semantic gap" in dialogue systems. By enabling efficient training directly on large action spaces, it removes the need for restrictive hand-coded abstractions. Future directions include combining this approach with supervised learning (pre-training on human-to-human transcripts) to further boost initial performance before reinforcement learning takes over.
Summary updated