Abstract
Spoken dialogue management involves complex decision-making under uncertainty, often modeled as Markov Decision Processes (MDPs). However, continuous state spaces and data scarcity pose significant challenges for traditional reinforcement learning (RL) methods. This paper introduces Fitted-Q iteration, a batch RL approach that utilizes function approximation to optimize dialogue policies. By learning from a fixed dataset of transitions, Fitted-Q achieves superior performance, greater stability, and higher computational efficiency compared to Least-Squares Policy Iteration (LSPI) and hand-coded baselines.
Introduction
The Dialogue Manager is the central component of a voice assistant, responsible for deciding the system's next action based on user input and context. Traditionally, these interactions are modeled using MDPs. Two primary hurdles complicate this task: the Curse of Dimensionality, where continuous states (e.g., speech recognition confidence) lead to an infinite state space, and Data Scarcity, as collecting human-computer dialogue is costly. This work addresses these issues by moving from "on-policy" learning to a "batch" approach that generalizes effectively from limited, fixed datasets.
Methodology
The proposed solution employs Fitted Value Iteration (FVI), specifically the Fitted-Q algorithm. Unlike step-by-step learning, Fitted-Q iterates over a fixed set of dialogue transitions $(s, a, r, s')$ to find an optimal policy.
To manage continuous state spaces, the $Q$-function is represented via Linear Function Approximation using Radial Basis Functions (RBFs). The model utilizes 351 features, consisting of 3 Gaussians per dimension across 13 possible actions. In each iteration, the algorithm updates its action-value estimates by solving a linear least-squares problem. Notably, Fitted-Q offers a computational complexity of $O(p^2)$ per iteration, where $p$ is the number of features, making it more efficient than the $O(p^3)$ complexity of LSPI.
Experimental Setup
The approach was evaluated in a simulated "form-filling" tourist information domain using the DIPPER dialogue framework. A "unigram" user simulator provided interactions. A corpus of 393,896 dialogue turns was generated using an $\epsilon$-greedy policy to ensure broad exploration of the problem space.
Key Results
Fitted-Q demonstrated significant advantages over both hand-coded policies and LSPI. It reached high performance levels with only a few thousand training samples, showcasing remarkable sample efficiency.
As illustrated in the figure above, Fitted-Q consistently outperformed the hand-coded baseline and maintained a lower standard deviation in rewards compared to LSPI. This indicates that the resulting dialogue manager is not only more effective but also more predictable and reliable. Furthermore, the algorithm proved to be computationally less intensive while providing marginally better rewards than LSPI.
Discussion and Future Work
A key insight is that Fitted-Q's use of regression allows it to interpolate actions for unseen states, which is critical for handling noisy recognition scores. Its off-policy nature enables learning from suboptimal data, and its lack of tunable hyperparameters simplifies practical deployment.
Future research will focus on automatic feature selection to replace manual RBF engineering and testing on real-world human corpora. Additionally, the authors aim to explore "online" RL integration using sample-efficient methods like Kalman Temporal Differences (KTD) to allow systems to learn in real-time during conversations.
Summary updated