Verena Rieser, Simon Keizer, Oliver Lemon, Xingkun Liu Proceedings of the 13th European Workshop on Natural Language Generation (ENLG), 2011.
Abstract
This paper explores the application of Reinforcement Learning (RL) to Information Presentation (IP) in spoken dialogue systems (SDS). Moving beyond rigid, hand-coded rules, the authors treat Natural Language Generation (NLG) as a statistical optimization problem. By framing IP as a Markov Decision Process, the researchers demonstrate that adaptive information delivery can improve objective task completion rates by up to 30% in real-world settings, even when faced with significant speech recognition noise.
Introduction
When a spoken dialogue system must present a set of results—such as a list of restaurants—it faces a strategic choice: should it summarize the options, compare specific candidates, or recommend a single top hit? Most current systems rely on "one-size-fits-all" hand-coded rules that are often too rigid to handle the inherent uncertainty of spoken language, such as misrecognitions or ambiguous user intent.
The authors argue that NLG should focus on "global" goals—helping the user successfully complete their task—rather than just "local" goals like sentence fluency. This paper presents an evaluation of a system that learns to adapt its presentation strategy based on the current state of the dialogue and the level of environmental noise.
Methodology
The researchers frame Information Presentation as a Markov Decision Process (MDP), allowing the system to learn an optimal policy through Reinforcement Learning.
The Framework and Strategies
The system makes incremental decisions at each turn, choosing between three primary strategies:
- SUMMARY: Providing an overview of the results (e.g., "There are 10 Indian restaurants...").
- COMPARE: Highlighting differences between specific items.
- RECOMMEND: Suggesting a specific entity that best matches the user's constraints.
Beyond selecting a strategy, the system also optimizes the number of attributes (such as price, location, or cuisine) to mention, ranging from one to five. The policy was trained using a hierarchical version of the SARSA algorithm (SHARSHA) in a simulated environment before being deployed for real-user evaluation. To ensure stability, the authors implemented a hybrid approach where the RL policy operated within certain rule-based constraints, such as ensuring every presentation ended with a recommendation to maintain dialogue flow.
Experimental Setup
The system was evaluated using two distinct user groups to create a robust dataset:
- Feb11-LOC: 304 dialogues from 19 local subjects (primarily university students).
- Feb11-AMT: 532 dialogues from 113 users recruited via Amazon Mechanical Turk.
The evaluation was conducted over VoIP, resulting in high Word Error Rates (WER) between 53% and 56%. The performance of the Trained IP (TIP) system was compared against a baseline (BASE) system that utilized a standard single-item recommendation strategy.
Key Results and Discussion
The evaluation revealed a significant advantage for the data-driven approach in terms of objective success.
Objective vs. Subjective Success
As shown in the results table, the TIP system consistently outperformed the BASE system in "Full Completion" (ObjSucc-FC), which required the user to find a venue and successfully retrieve specific details like a phone number or address. In the pooled data (Feb11-TOT), full completion rose from 34.78% to 42.98%.
Interestingly, the study found a "subjective gap." While the TIP system was objectively more successful, users did not necessarily rate its phrasing or understanding higher than the baseline. This suggests that while more informative prompts help users reach their goals, they may be perceived as more demanding or less "natural" by the users.
Learned Optimal Strategies
Analysis of the optimized policy showed that the most frequent strategy was a combination: SUMMARY(2) + RECOMMEND(2). This indicates that providing a brief summary of two attributes followed by a recommendation of two attributes is the most effective way to guide a user toward task completion in a noisy environment.
Conclusion
This research demonstrates that Reinforcement Learning can effectively manage the trade-off between being informative and being brief in spoken dialogue. By adapting to the dialogue context and noise levels, the TIP system provided a 30.1% relative increase in task success for the Mechanical Turk group. Future work will focus on the joint optimization of the Dialogue Manager and the NLG component to further refine system performance and user satisfaction.
Summary updated