Abstract
The optimization of spoken dialog management policies is a non-trivial task due to the erroneous inputs from speech recognition and language understanding modules. The dialog manager needs to ground uncertain semantic information at times to fully understand the need of human users and successfully complete the required dialog tasks. Approaches based on reinforcement learning are currently mainstream in academia and have been proved to be effective, especially when operating in noisy environments. However, in reinforcement learning the dialog strategy is often represented by complex numeric model and thus is incomprehensible to humans. The trained policies are very difficult for dialog system designers to verify or modify, which largely limits the deployment for commercial applications. In this paper we propose a novel framework for optimizing dialog policies specified in human-readable domain language using genetic algorithm. We present learning algorithms using user simulator and real human-machine dialog corpora. Empirical experimental results show that the proposed approach can achieve competitive performance on par with some state-of-the-art reinforcement learning algorithms, while maintaining a comprehensible policy structure.
Introduction
Spoken Dialog Systems (SDS) are the foundational technology for voice assistants and automated interfaces. A critical component of these systems is the Dialog Manager (DM), which serves as the "brain," deciding the system's next action based on potentially noisy inputs from speech recognition (ASR) and language understanding (SLU) modules.
Traditionally, developers have relied on hand-crafted rules, which are interpretable but often brittle in the face of environmental noise. Conversely, Reinforcement Learning (RL) has emerged as a powerful tool for creating robust policies, yet it often results in "black box" models that are difficult for engineers to verify or modify. This research addresses this gap by proposing GA-DM, a framework that uses Genetic Algorithms (GA) to optimize human-readable rule templates.
Methodology
The GA-DM framework treats dialog policy optimization as an evolutionary process, focusing on tuning the parameters of human-defined rule templates.
Policy Templates and Genetic Optimization
Instead of learning a policy from scratch, the system utilizes templates defined by a Backus-Naur Form (BNF) grammar. These templates consist of "if-then-else" logic, such as: "If the confidence score for a slot is less than $\theta_1$, then ask for explicit confirmation." The free parameters (e.g., $\theta_1$) are treated as "genes" within a population. Through selection, crossover, and mutation, the GA evolves these parameters to find optimal configurations.
Fitness Evaluation
Two primary methods were proposed to evaluate policy fitness:
- User Simulation: Policies are tested against a simulated user to measure average reward, task success rate, and dialog length.
- Q-Points Regression: For static datasets (corpora), a novel batch method uses Fitted Q-Iteration to estimate action values, rewarding GA individuals whose decisions align with high-value actions.
The approach was evaluated using a simulated restaurant information domain and the public DSTC2 (Dialog State Tracking Challenge 2) dataset.
Results and Discussion
Experimental results demonstrate that GA-DM achieves state-of-the-art performance while maintaining transparency.
As shown in the performance metrics, GA-DM (represented by green triangles) significantly outperforms both standard rule-based heuristics and RL-based systems (Q-learning with linear approximation) in high-noise environments where semantic error rates exceed 0.6. Specifically, GA-DM maintains higher task completion rates and average rewards as noise increases.
On the real-world DSTC2 dataset, the GA-DM framework achieved a testing reward of 129.29, surpassing the supervised learning baseline (114.39) and standard thresholded Q-learning (122.21). The primary advantage of GA-DM is its interpretability; the final policy remains a set of human-readable rules with optimized numerical thresholds, allowing for direct verification by system designers.
Conclusion
The GA-DM framework successfully bridges the gap between interpretable rule-based systems and high-performance reinforcement learning. By constraining the search space to human-defined templates, the system achieves rapid optimization and robust performance in noisy environments. Future work will explore structural optimization—allowing the GA to evolve the logic branches of the rules themselves—and further reducing the computational overhead associated with evolutionary evaluations.
Reference: Ren, H., Zhao, Q., & Yan, Y. (2016). Policy Optimization for Spoken Dialog Management Using Genetic Algorithm. IEICE Transactions on Information and Systems, E99-D(10), 2499-2507.
Summary updated