Introduction
Designing effective Spoken Dialogue Systems (SDS) remains a significant challenge due to the high complexity of state spaces and the need for specialized expertise in Reinforcement Learning (RL). While Markov Decision Processes (MDPs) are a standard academic approach, industry systems often rely on handcrafted finite state machines to ensure predictable behavior. However, optimizing these handcrafted designs often violates the Markovian assumption—where the current state must contain all information needed for future predictions—causing standard RL algorithms to fail.
Methodology
This research proposes a hybrid framework called the Module-Variable Decision Process (MVDP) paired with Compliance Based Reinforcement Learning (CBRL). MVDP decomposes the dialogue into discrete modules (e.g., greeting, troubleshooting), each with its own local context and design alternatives. CBRL addresses the credit assignment problem in non-Markovian environments by weighting updates based on how "compliant" a dialogue episode was with the current policy. This allows the system to learn optimal design-level decisions—such as choosing between different text-to-speech styles or question orders—without requiring the AI to manage every low-level dialogue move.
Experimental Setup
The approach was validated through a proof-of-concept study involving a Livebox (DSL box) installation support task. The study included 40 subjects performing four scenarios each, resulting in a corpus of 160 dialogues in French. The system was configured with four specific "design points" where the RL algorithm could select between pre-defined strategies.
Results and Discussion
The experimental results demonstrate that learning at the design level leads to rapid performance improvements. The system showed significant gains within 40 to 70 dialogues, bypassing the slow convergence typical of large-scale MDPs.
As shown in the reward evolution, the system successfully transitioned from exploratory trial-and-error to an optimized strategy. This optimization translated directly into a better user experience, specifically reducing misunderstandings by an average of one per dialogue.
Furthermore, the system learned interaction styles that were more robust to environmental factors. As illustrated in the error analysis, "ASR noise" (errors from environmental noise or user manipulation) was effectively halved by the end of the experiment. This demonstrates the reliability of the handcrafted design when augmented with machine learning optimization.
Conclusion
By applying RL to high-level design choices rather than low-level dialogue moves, this framework achieves the reliability of handcrafted systems with the optimization benefits of machine learning. The results are highly interpretable, allowing designers to understand which strategies perform best and apply these insights to future VUI designs. Future work will focus on scaling this approach to more complex design points and integrating real-time performance monitoring into design tools.
Summary updated