Abstract: Ivana Kruijff-Korbayová, Nate Blaylock, Ciprian Gerstenberger, Verena Rieser, Tilman Becker, Michael Kaisser, Peter Poller, Jan Schehl. Proceedings of the Tenth European Workshop on Natural Language Generation (ENLG-05).
2005.
Introduction
Designing a system that can both speak to a user and display information on a screen presents a significant "fission" problem: how does the system decide which modality to use for specific pieces of information? Most existing multimodal datasets are either human-to-human, which lacks the constraints of a computer interface, or use highly scripted Wizard-of-Oz (WoZ) setups where system responses are pre-determined.
This research addresses the lack of data regarding adaptive output planning. The goal is to understand how a system should adjust its behavior based on the user's context—specifically, whether they are focused on a task like driving or are in a stationary, low-cognitive-load environment.
Experimental Methodology
The researchers developed a sophisticated, distributed WoZ environment to collect the SAMMIE-2 corpus. This setup involved five distinct roles per session: a user, a wizard (acting as the system), an experiment leader, and two transcribers.
Technical Innovations
To ensure the data collected was both natural and useful for training future AI, several technical innovations were implemented:
- Unscripted Wizardry: Unlike previous studies, wizards were not bound by a script. They were free to respond naturally, choosing to speak, display information on a screen, or combine both modalities.
- Screen Output Planning Module: To prevent the wizard from being overwhelmed by UI design tasks in real-time, the system automatically generated four different visual layout options (e.g., a full table of results vs. a simple text summary) based on the database query. The wizard simply selected the version they deemed most appropriate for the current context.
- Simulated Imperfection: To maintain the illusion of interacting with an AI, user speech was transcribed and then "corrupted" (by randomly deleting words) before reaching the wizard. Conversely, the wizard’s typed responses were converted into synthetic speech for the user.
- Dual-Task Environment: Users performed music-related tasks (searching for songs, building playlists) while either stationary or using a "Lane Change" driving simulator to measure the impact of cognitive load on modality preference.
The SAMMIE-2 Corpus
The primary output of this study is the SAMMIE-2 corpus, which contains data from 24 subjects performing four tasks each. The dataset includes audio/video recordings and detailed logfiles of every system event, database query, and modality choice. To ensure realism, wizards queried the FreeDB database, containing nearly 1 million songs.
Key Insights and Results
The study provided several critical observations regarding multimodal interaction:
- Modality Adaptation: Context is the primary driver of modality choice. Wizards naturally shifted toward speech-only output when the user was driving. Users reported that while screen output was helpful for "memory assistance" (e.g., verifying a long playlist), it was highly distracting during high-stress driving.
- Visual Preference as Default: When presenting search results in a stationary context, wizards chose to display a full table 78.6% of the time, suggesting high-density visual information is the preferred default when a screen is available.
- Driving Interference: 85.7% of subjects found screen output distracting while driving, and 76.2% explicitly stated a preference for more verbal feedback in that context.
- Information Granularity: Speech and screen were rarely used redundantly. Wizards often used the screen for high-density data (like a list of 10 albums) while using speech to provide a high-level summary ("I found 10 albums by Prince").
- User Satisfaction: There was a strong correlation (Spearman’s r = 0.658) between user satisfaction and the specific multimodal strategy employed, proving that how information is presented is as vital as the information itself.
Conclusion and Future Work
While the human-in-the-loop pipeline introduced some latency issues, the SAMMIE-2 corpus provides a "gold standard" of expert labels for multimodal behavior. Future work will focus on using this data to train reinforcement learning models for automated output planning and replacing human transcribers with Automatic Speech Recognition (ASR) to test system robustness against real-world noise.
Summary updated