Anti-Overestimation Dialogue Policy Learning for Task-Completion Dialogue System

Chang Tian, Wenpeng Yin, Marie-Francine Moens

Abstract

Reinforcement Learning (RL) based dialogue policies often suffer from overestimation bias, leading to suboptimal decision-making and task failure. We present Dynamic Partial Average (DPAV), a novel estimator that mitigates this bias by incorporating a weighted average of maximum and minimum action values. DPAV achieves state-of-the-art performance on task-completion benchmarks with significantly lower computational overhead than ensemble-based methods.

Introduction

In task-oriented dialogue systems, the Dialogue Policy Learning (DPL) component is responsible for selecting system actions that lead to successful task completion. Modern DPL frameworks frequently employ Deep Q-Networks (DQN) to map conversation states to optimal actions. However, the inherent overestimation bias in Q-learning—where the max operator favors noisy, inflated value estimates—remains a critical bottleneck.

The Dialogue Policy Learning module within a task-oriented system architecture, highlighting the discrepancy between predicted values and ground truth.

The Overestimation Problem

Overestimation bias occurs when the agent becomes overly optimistic about certain actions due to approximation errors. In dialogue management, this often manifests as repetitive questioning or the selection of irrelevant intents. For instance, an agent might repeatedly ask for the number of tickets even after the user has provided that information, simply because the estimated value for that action remains erroneously high. This leads to failed simulation episodes and poor user experience.

Methodology: Dynamic Partial Average (DPAV)

The proposed Dynamic Partial Average (DPAV) estimator addresses this by "anchoring" the maximum predicted value with the minimum predicted value. The target Q-value is calculated as:

$$Q_*(s_{t+1}) \approx (1 - \lambda_t) \cdot \max_{a'} Q(s_{t+1}, a') + \lambda_t \cdot \min_{a'} Q(s_{t+1}, a')$$

The parameter $\lambda_t$ is a dynamic weight that decays over the course of training. Initially, a higher $\lambda$ aggressively penalizes potential overestimation. As the model converges and its estimates become more reliable, $\lambda$ decreases, allowing the policy to focus on the most promising actions without excessive pessimism. Unlike ensemble methods that require multiple parallel networks, DPAV operates within a single-model architecture, maintaining $O(N)$ complexity.

Experimental Results

The DPAV DQN was evaluated across three benchmarks: Movie-ticket booking (280 dialogues), Restaurant reservation (4,103 dialogues), and Taxi ordering (3,094 dialogues). These datasets vary in complexity, requiring the agent to track between 16 and 30 distinct information "slots."

Comparison between an overestimation-prone policy (left) and the anti-overestimation DPAV policy (right). The standard policy gets stuck in a loop, while DPAV successfully completes the task.

DPAV consistently achieved higher success rates and average rewards compared to standard DQN and Double DQN. In the Taxi domain, DPAV demonstrated significantly faster convergence. Crucially, visualization of Q-values confirmed that DPAV estimates remained closely aligned with actual rewards, whereas standard DQN values were significantly inflated. Furthermore, DPAV matched the performance of heavy ensemble methods like SUNRISE while requiring a fraction of the computational load.

Conclusion

DPAV provides an efficient and effective mechanism for reducing overestimation bias in dialogue policy learning. By balancing optimistic exploration with a dynamic value anchor, it ensures more robust task completion and practical scalability for real-world conversational agents.

Summary updated

Abstract

A dialogue policy module is an essential part of task-completion dialogue systems. Recently, increasing interest has focused on reinforcement learning (RL)-based dialogue policy. Its favorable performance and wise action decisions rely on an accurate estimation of action values. The overestimation problem is a widely known issue of RL since its estimate of the maximum action value is larger than the ground truth, which results in an unstable learning process and suboptimal policy. This problem is detrimental to RL-based dialogue policy learning. To mitigate this problem, this paper proposes a dynamic partial average estimator (DPAV) of the ground truth maximum action value. DPAV calculates the partial average between the predicted maximum action value and minimum action value, where the weights are dynamically adaptive and problem-dependent. We incorporate DPAV into a deep Q-network as the dialogue policy and show that our method can achieve better or comparable results compared to top baselines on three dialogue datasets of different domains with a lower computational load. In addition, we also theoretically prove the convergence and derive the upper and lower bounds of the bias compared with those of other methods.
BibTeX
@inproceedings{tian2022anti,
  author = {Chang Tian and Wenpeng Yin and Marie-Francine Moens},
  title = {{Anti-Overestimation Dialogue Policy Learning for Task-Completion Dialogue System}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.findings-naacl.43},
}
Figure