QuickSet: Multimodal Interaction for Simulation Set-up and Control

Philip R. Cohen, Michael Johnston, David McGee, Sharon L. Oviatt, Jay Pittman, Ira A. Smith, Liang Chen, Josh ClowANLP 1997 · OpenReview

TL;DR

This paper introduces QuickSet, a novel multimodal system that combines pen and voice input on a handheld PC for setting up and controlling large-scale distributed military simulations. It addresses the limitations of traditional GUIs in complex simulation environments by offering a more intuitive and efficient interaction paradigm.

What Problem Does This Paper Address?

The paper tackles the challenge of effectively interacting with increasingly complex and large-scale distributed interactive simulations (DIS), particularly in military training. Traditional graphical user interfaces (GUIs) struggle with the sheer number of entities and options required for scenario setup, especially when constrained by smaller screen sizes on mobile devices like PDAs. The authors argue that while GUIs excel at reminding users of options, they are weak in handling numerous objects or those not currently visible. Conversely, linguistically-based interfaces, especially spoken language, can describe large sets of objects but are often suboptimal for spatial tasks. This creates a need for a more robust interaction method that leverages the strengths of both modalities.

Methods

QuickSet employs a multimodal interaction approach, integrating pen-based gestures and spoken language. The system runs on handheld PCs and communicates with simulation environments like NRaD's LeatherNet and ModSAF through a distributed agent architecture based on the Open Agent Architecture (OAA).

Key components include:

  • QuickSet Interface: A geo-referenced map on the handheld PC that displays entities and allows users to pan, zoom, draw directly on the map, create entities, define behaviors, and observe the simulation.
  • Speech Recognition Agent: Utilizes IBM's VoiceType technology for continuous, speaker-independent speech recognition with a vocabulary of 450 words.
  • Gesture Recognition Agent: Processes pen input, combining results from HMM and neural net recognizers to recognize up to 45 gestures, which translate into military symbols, shapes, and lines.
  • Natural Language Agent: Processes spoken input using a definite clause grammar to produce typed feature structures representing utterance meaning.
  • Multimodal Integration Agent: This is the core of the system's novelty. It takes typed feature structures from both speech and gesture agents and produces a unified interpretation. This agent aims to compensate for the weaknesses of individual unimodal recognizers by selecting the highest-scoring combined interpretation, ensuring that the communicative modalities support each other.

Datasets

Not discussed in the paper. The paper focuses on the system's design and application rather than empirical evaluation on specific datasets.

Key Insights and Ideas

  • Multimodal interaction is superior for complex spatial tasks: The paper strongly advocates for the benefits of combining pen and voice input, especially for map-based interactions, citing empirical results that demonstrate clear advantages over speech-only input.
  • Compensatory nature of modalities: The multimodal integration agent is designed to leverage the strengths of each modality to overcome the weaknesses of the other. For instance, a gesture can specify a location, making the accompanying speech more concise and less prone to errors.
  • Agent-based architecture for flexibility: The use of OAA allows QuickSet to interoperate with various legacy applications and adapt to different hardware configurations, from standalone handhelds to distributed networks.
  • Bridging the gap between small screens and complex data: QuickSet's approach is a direct response to the trend of smaller devices and the increasing complexity of simulations, offering a more manageable interaction paradigm.

Key Results

The paper illustrates QuickSet's functionality through examples:

  • Users can create entities by speaking and gesturing simultaneously (e.g., "red T72 platoon" while pointing at a location).
  • Complex drawing tasks, like creating a "phase line" between two grid coordinates, are simplified from a lengthy spoken command to a spoken phrase combined with drawing the line.
  • The system has been successfully delivered and used by the US Navy and US Marine Corps for training scenario setup and control.
  • It was subjected to an extreme noise environment during a US Army exercise, where spoken interaction was infeasible, but users successfully used gestural input. This led to the development of complete unimodal pen and speech functionality.

Limitations and Future Work

  • Unimodal functionality: While the system supports unimodal speech and pen input, the paper notes that the initial focus was on multimodal interaction, and the complete overlap in functionality was a later development driven by real-world usage.
  • Vocabulary size: The speech recognition agent currently has a vocabulary of 450 words, which might be a limitation for more complex commands.
  • Future work: The authors mention ongoing research into further developing QuickSet's capabilities and quantifying the general benefits of multimodal interaction. They also highlight its extension for the ExInit system to create division-sized exercises.

Summary updated

BibTeX
@inproceedings{cohen1997quickset,
  author = {Philip R. Cohen and Michael Johnston and David McGee and Sharon L. Oviatt and Jay Pittman and Ira A. Smith and Liang Chen and Josh Clow},
  title = {{QuickSet: Multimodal Interaction for Simulation Set-up and Control}},
  booktitle = {ANLP},
  year = {1997},
  url = {https://openreview.net/forum?id=JC9PPFqpUH},
}
Figure