Abstract
We investigate the use of machine learning in combination with feature engineering techniques to explore human multimodal clarification strategies and the use of those strategies for dialogue systems. We learn from data collected in a Wizard-of-Oz study where different wizards could decide whether to ask a clarification request in a multimodal manner or else use speech alone. We show that there is a uniform strategy across wizards which is based on multiple features in the context. These are generic runtime features which can be implemented in dialogue systems. Our prediction models achieve a weighted f-score of 85.3% (which is a 25.5% improvement over a one-rule baseline). To assess the effects of models, feature discretisation, and selection, we also conduct a regression analysis. We then interpret and discuss the use of the learnt strategy for dialogue systems. Throughout the investigation we discuss the issues arising from using small initial Wizard-of-Oz data sets, and we show that feature engineering is an essential step when learning from such limited data.
Introduction
In multimodal dialogue systems, such as in-car infotainment or smart displays, the system must frequently decide how to present information. When a user's intent is ambiguous, the system issues a clarification request (CR). It can do so using speech alone or by supplementing speech with a visual display (e.g., a list of options). While visual aids can reduce cognitive load, they can also be distracting, particularly in safety-critical environments like driving.
This research explores whether human "wizards"—acting as the system in a controlled study—follow a systematic, intuitive strategy when choosing between unimodal (speech-only) and multimodal (speech + graphics) clarification. By decoding these human strategies using machine learning, we can develop more natural and effective automated dialogue systems.
Methodology and Feature Engineering
The researchers treated the choice of output modality as a supervised learning problem: predicting whether the system should output a graphic (graphic-yes) or not (graphic-no) based on the current conversational state.
A significant challenge in dialogue research is the scarcity of high-quality data. To address this, the study employed a rigorous experimental pipeline centered on feature engineering.
The process involved four key stages:
- Feature Extraction: 18 features were extracted across three categories: Local (current turn info), Dialogue History (previous interactions), and User Model (user habits and state, such as driving).
- Discretization: Continuous numerical values were converted into ranges (e.g., "more than one match") to help models handle sparse data more effectively.
- Feature Selection: Algorithms like Selective Naive Bayes were used to identify the most predictive features, reducing noise.
- Classification: Several models, including Naive Bayes, Decision Trees (J4.8), and Rule Induction (JRIP), were trained and evaluated.
Data Collection: The Wizard-of-Oz Corpus
The study utilized a Wizard-of-Oz (WOZ) corpus of German task-oriented dialogues for an in-car music player. The dataset included 70 dialogues and 1,772 turns. From this, 152 specific instances were identified where a human wizard had to issue a clarification request. During the data collection, users sometimes performed a primary driving task in a simulator, providing a realistic context for multimodal interaction.
Key Results and the Learned Strategy
The experimental results demonstrated that human wizards are remarkably consistent in their decision-making. The best-performing model—Naive Bayes combined with specific feature selection and discretization—achieved a weighted f-score of 85.3%, representing a 25.5% improvement over a simple "one-rule" baseline.
The study revealed that the "stickiness" of multimodality is a primary driver: if a user has already engaged with the screen (e.g., by clicking or referencing it), wizards are significantly more likely to continue using the screen for clarifications.
As shown in the decision tree above, the learned strategy can be distilled into a clear logic. For instance, if the user has interacted with the screen before (screenUser > 1) and the screen was used recently (screenHist > 0.5), the system should almost certainly provide a graphic. Conversely, if the user has not interacted with the screen, the system defaults to speech-only.
Discussion and Future Work
This research highlights that even with small datasets, machine learning can successfully decode complex human interaction strategies provided that robust feature engineering is applied. The consistency found across different wizards suggests that these strategies are not arbitrary but are grounded in the conversational context.
However, the study also notes limitations. The sample size of 152 instances is small, and the strategies learned are specific to the in-car music domain. Furthermore, supervised learning focuses on mimicking human behavior, which may not always be optimal. Future work should explore Reinforcement Learning (RL) to refine these human-like strategies, optimizing them for specific outcomes like user satisfaction or task efficiency across diverse multimodal domains.
Summary updated