Abstract
Reinforcement Learning (RL) based dialogue policies often suffer from overestimation bias, leading to suboptimal decision-making and task failure. We present Dynamic Partial Average (DPAV), a novel estimator that mitigates this bias by incorporating a weighted average of maximum and minimum action values. DPAV achieves state-of-the-art performance on task-completion benchmarks with significantly lower computational overhead than ensemble-based methods.
Introduction
In task-oriented dialogue systems, the Dialogue Policy Learning (DPL) component is responsible for selecting system actions that lead to successful task completion. Modern DPL frameworks frequently employ Deep Q-Networks (DQN) to map conversation states to optimal actions. However, the inherent overestimation bias in Q-learning—where the max operator favors noisy, inflated value estimates—remains a critical bottleneck.
The Overestimation Problem
Overestimation bias occurs when the agent becomes overly optimistic about certain actions due to approximation errors. In dialogue management, this often manifests as repetitive questioning or the selection of irrelevant intents. For instance, an agent might repeatedly ask for the number of tickets even after the user has provided that information, simply because the estimated value for that action remains erroneously high. This leads to failed simulation episodes and poor user experience.
Methodology: Dynamic Partial Average (DPAV)
The proposed Dynamic Partial Average (DPAV) estimator addresses this by "anchoring" the maximum predicted value with the minimum predicted value. The target Q-value is calculated as:
$$Q_*(s_{t+1}) \approx (1 - \lambda_t) \cdot \max_{a'} Q(s_{t+1}, a') + \lambda_t \cdot \min_{a'} Q(s_{t+1}, a')$$
The parameter $\lambda_t$ is a dynamic weight that decays over the course of training. Initially, a higher $\lambda$ aggressively penalizes potential overestimation. As the model converges and its estimates become more reliable, $\lambda$ decreases, allowing the policy to focus on the most promising actions without excessive pessimism. Unlike ensemble methods that require multiple parallel networks, DPAV operates within a single-model architecture, maintaining $O(N)$ complexity.
Experimental Results
The DPAV DQN was evaluated across three benchmarks: Movie-ticket booking (280 dialogues), Restaurant reservation (4,103 dialogues), and Taxi ordering (3,094 dialogues). These datasets vary in complexity, requiring the agent to track between 16 and 30 distinct information "slots."
DPAV consistently achieved higher success rates and average rewards compared to standard DQN and Double DQN. In the Taxi domain, DPAV demonstrated significantly faster convergence. Crucially, visualization of Q-values confirmed that DPAV estimates remained closely aligned with actual rewards, whereas standard DQN values were significantly inflated. Furthermore, DPAV matched the performance of heavy ensemble methods like SUNRISE while requiring a fraction of the computational load.
Conclusion
DPAV provides an efficient and effective mechanism for reducing overestimation bias in dialogue policy learning. By balancing optimistic exploration with a dynamic value anchor, it ensures more robust task completion and practical scalability for real-world conversational agents.
Summary updated