Deep Reinforcement Learning for On-line Dialogue State Tracking

Zhi Chen, Chen Lü, Xiang Zhou, Kai YuCommunications 2023 · DOI · arXiv · OpenAlex

Abstract

Traditional Dialogue State Tracking (DST) models are typically trained offline using static, human-labeled datasets, leading to brittleness in dynamic environments and a decoupling from the dialogue policy. This paper presents a novel framework that treats DST as an on-line optimization problem using Deep Reinforcement Learning (DRL). By introducing a "companion teaching" method, where a pre-trained teacher model guides a Deep Deterministic Policy Gradient (DDPG) student tracker, the system can adapt its understanding of user intent through live interaction. Experimental results on the DSTC2 and DSTC3 benchmarks demonstrate significant improvements in success rates and dialogue efficiency compared to traditional rule-based and supervised baselines.

Introduction: The Challenge of Static Tracking

In task-oriented spoken dialogue systems, the Dialogue State Tracker (DST) acts as the system's memory, estimating the "belief state"—the most probable user intent—from noisy audio inputs. Historically, DST has been framed as a supervised learning task. While effective in controlled settings, this approach suffers from three critical limitations: human labels are expensive and unavailable during live deployment; models often fail to generalize to new user behaviors; and the tracker is trained independently of the decision-making policy, ignoring how tracking errors impact conversational success.

To address these issues, we propose an on-line learning framework where the DST module optimizes itself through reinforcement learning, allowing for continuous adaptation and joint optimization with the dialogue policy.

The Companion Teaching Framework

Optimizing a tracker via RL is challenging because the action space (the belief state) is continuous and high-dimensional. A naive RL agent exploring this space from scratch would likely lead to catastrophic system failure before learning useful patterns.

The Companion Teaching DRL framework for on-line dialogue state tracking.

As shown in the framework above, we implement a Companion Teaching architecture. The core components include:

  1. The Tracking Agent: A DDPG-based agent that outputs continuous belief state vectors.
  2. The Teacher (Auxiliary DST): A pre-trained, rule-based model that provides a reference belief state ($b^a$) to guide the student.
  3. Hybrid Reward Function: The agent receives rewards based on both task success and a "Basic Score" that measures proximity to the teacher's estimate. This prevents the agent from entering nonsensical states during early training phases.

Sub-agent Decomposition

To manage complexity, the tracker is decomposed into specialized sub-agents for different slot types: Goals (e.g., food type), Requests (e.g., phone number), and Methods (e.g., by constraints).

Decomposition of the Tracking Agent into sub-agents (TA_G, TA_R, TA_M) using an Actor-Critic (DDPG) architecture.

This modularity allows the system to focus optimization on high-impact areas. For instance, accurately tracking Goals is often more critical for task completion than tracking individual Requests.

Experimental Evaluation

We evaluated the framework using the DSTC2 and DSTC3 benchmarks, employing an agenda-based user simulator to provide the thousands of interactions required for RL.

Key Results

The DRL-based tracking agents significantly outperformed baseline polynomial trackers:

  • Success Rate: On the more challenging DSTC3 dataset, the joint training approach achieved a success rate of 0.800, a notable improvement over the 0.754 baseline.
  • Dialogue Efficiency: The system learned to resolve tasks faster. In DSTC3, the average dialogue length dropped from 6.58 turns to 4.31 turns.
  • Robustness to Noise: The performance gains were more pronounced in DSTC3 than in DSTC2, suggesting that the DRL approach is particularly adept at handling high semantic error rates from speech recognizers.

Discussion and Insights

A critical finding was the necessity of the teacher model; systems without "companion teaching" failed to converge, proving that "cold-starting" a tracker in a complex dialogue environment is currently unfeasible without supervised guidance. Furthermore, we observed mutual adaptation: when the tracker and policy were trained jointly, the policy learned to compensate for specific tracking quirks, leading to higher overall success than optimizing either component in isolation.

Conclusion

This work demonstrates that moving DST from static supervision to on-line DRL is not only possible but highly beneficial for system performance and robustness. While current results rely on simulators, this framework paves the way for truly adaptive dialogue systems that learn directly from human interaction. Future research will focus on annealing the teacher's influence as the student matures and transitioning these models to real-world human-in-the-loop environments.

Summary updated

BibTeX
@inproceedings{chen2023deep,
  author = {Zhi Chen and Chen Lü and Xiang ZHOU and Kai Yu},
  title = {{Deep Reinforcement Learning for On-line Dialogue State Tracking}},
  booktitle = {Communications},
  year = {2023},
  doi = {10.1007/978-981-99-2401-1\_25},
}
Figure