Abstract
We present and evaluate a novel approach to natural language generation (NLG) in statistical spoken dialogue systems (SDS) using a data-driven statistical optimization framework for incremental information presentation (IP), where there is a trade-off to be solved between presenting “enough" information to the user while keeping the utterances short and understandable. The trained IP model is adaptive to variation from the current generation context (e.g. a user and a non-deterministic sentence planner), and it incrementally adapts the IP policy at the turn level. Reinforcement learning is used to automatically optimize the IP policy with respect to a data-driven objective function. In a case study on presenting restaurant information, we show that an optimized IP strategy trained on Wizard-of-Oz data outperforms a baseline mimicking the wizard behavior in terms of total reward gained. The policy is then also tested with real users, and improves on a conventional hand-coded IP strategy used in a deployed SDS in terms of overall task success. The evaluation found that the trained IP strategy significantly improves dialogue task completion for real users, with up to a 8.2% increase in task success. This methodology also provides new insights into the nature of the IP problem, which has previously been treated as a module following dialogue management with no access to lower-level context features.
Introduction
In Spoken Dialogue Systems (SDS), the "Information Presentation" (IP) phase is a delicate balancing act. If the system provides too little information, the user is left guessing; if it provides too much, the user becomes confused or impatient. Traditionally, NLG is treated as a "one-shot" module that follows the Dialogue Manager. However, this ignores the inherent uncertainty in the pipeline, such as speech recognition errors or varying text-to-speech quality. This paper addresses the need for an NLG component that can adapt incrementally to these low-level context features and uncertainties to maximize the chances of a successful conversation.
Methodology: NLG as Sequential Planning
The core innovation of this work is treating NLG as a Markov Decision Process (MDP). Instead of generating a single static response, the system plans a sequence of generation actions—such as summarizing results, recommending a top item, or deciding to stop and listen to the user.
To optimize this process, the authors employed Reinforcement Learning (RL), specifically a hierarchical SARSA algorithm with linear function approximation. The system learns an optimal policy by interacting with a simulated environment. To ensure realism, the training environment was bootstrapped using Wizard-of-Oz (WoZ) data, where human "wizards" acted as the system. Furthermore, a data-driven reward function was derived using stepwise linear regression on the WoZ data to identify features—like sentence length and database hit counts—that correlate most strongly with user satisfaction.
Information Presentation Strategies
The system's state space includes features like the number of attributes filled (price, cuisine, etc.), the number of sentences generated, and the number of matching results in the database. Based on these, the system selects from several high-level strategies to present information effectively.
These strategies allow the system to adapt to the volume of information. For instance, if there are many database hits, it might start with a summary. If only a few items remain, it might transition to a comparison or a direct recommendation. This hierarchical approach keeps the state space manageable while allowing for complex, multi-step utterances.
Experimental Evaluation and Results
The proposed model was evaluated across three primary datasets: a WoZ dataset of 213 dialogues, the LOC corpus (304 dialogues with local subjects), and the AMT corpus (532 dialogues via Amazon Mechanical Turk). The domain focused on a restaurant search task in Edinburgh involving 340 venues.
The results demonstrate that the RL-trained policy (TIP) significantly outperforms traditional hand-coded strategies. Key findings include:
- Task Success: The TIP system achieved up to an 8.2% absolute increase in full task success compared to standard IP strategies.
- Efficiency: While optimized prompts were sometimes longer, they were more effective, leading to fewer overall dialogue turns to reach a successful conclusion.
- Adaptability: The system proved capable of "listening" to lower-level modules, adjusting its strategy based on predicted TTS quality and actual sentence length—features typically ignored by traditional NLG planners.
Conclusion and Future Work
This research demonstrates that reimagining NLG as a sequential decision-making problem under uncertainty leads to more robust and successful dialogue systems. By moving away from static rules toward an adaptive, RL-optimized policy, the system better manages the trade-off between information density and user comprehension.
Future work aims to address the current decoupling between the Dialogue Manager and the NLG module. Jointly optimizing these components could prevent mismatches and further improve performance. Additionally, the authors plan to explore generation under broader uncertainties, such as dynamic user goals and environmental context like user gaze or location.
Summary updated