The CommandTalk Spoken Dialogue System

Amanda Stent, John Doweling, Jean Mark Gawron, Elizabeth Owen Bratt, Robert C. MooreACL 1999 · DOI

Introduction

Interacting with complex battlefield simulations like ModSAF (Modular Semi-Automated Forces) is a cognitively demanding task. Traditional graphical user interfaces (GUIs) can be cumbersome in high-pressure environments, while basic voice-command systems often lack the flexibility to handle human errors, changes of mind, or situational ambiguities. CommandTalk addresses these challenges by providing a sophisticated spoken dialogue interface that supports mixed-initiative interactions, multi-modal gestures, and real-time adaptation to a dynamic simulated world.

System Architecture

CommandTalk is built upon SRI’s Open Agent Architecture (OAA), a distributed framework where independent agents—specializing in speech recognition, natural language processing, and simulation control—cooperate to fulfill user requests.

A cornerstone of the system is the Gemini unified grammar. Unlike systems that utilize separate models for recognition and generation, CommandTalk uses a single unification-based grammar for both. This ensures linguistic consistency; the system’s responses naturally "prime" the user to use vocabulary and structures that the system is already configured to understand.

Dialogue Management and the "Guard" Mechanism

To manage the inherent messiness of human conversation, CommandTalk employs a Dialogue Stack to track open threads and a Dialogue Trail to record a history of all stack operations. This history allows the system to "undo" state changes if a user correction reveals that a previous sub-dialogue was misinterpreted.

One of the system's most innovative features is the "Guard" mechanism. Guards are procedural tests that verify if a command is executable within the current state of the simulation. If a prerequisite is missing—such as a target objective that has not yet been defined—the system does not simply return an error. Instead, it triggers a sub-dialogue to help the user resolve the issue.

The guard mechanism triggering a sub-dialogue to create a missing objective.

Multimodal Integration and Reference Resolution

CommandTalk treats physical actions, such as mouse clicks on a map, as first-class linguistic citizens. Clicks are resolved into coordinates and integrated into the semantic representation of the dialogue. To resolve pronouns and deictic expressions (like "here" or "that unit"), the system maintains two distinct lists:

  1. Salience List: Objects currently visible or recently created in the simulation.
  2. Focus Spaces: Linguistic items explicitly mentioned in the conversation.

Handling Ambiguity and Corrections

The robustness of the system is demonstrated through its ability to handle complex dialogue patterns, including ambiguity resolution and corrections. When a user provides an underspecified command, the system identifies the ambiguity and asks for clarification. Similarly, if a user realizes they made a mistake, they can issue a correction (e.g., "I meant Alpha") which the system processes by referencing the dialogue trail to update the intended action.

Example of ambiguity resolution and user correction within a single interaction.

Evaluation and Future Directions

Formal quantitative evaluation of CommandTalk remains a challenge due to the classified nature of the military environments in which it is deployed. Current insights are drawn from qualitative performance in complex scenarios, such as handling delayed responses when a user asks a question in the middle of a multi-step command sequence.

Future work focuses on increasing domain independence. The team is currently porting the dialogue manager to the air travel planning domain (ATIS) to utilize public datasets for more rigorous evaluation and to move toward a more generalized representation of actions using explicit plan models.

Summary updated

BibTeX
@inproceedings{stent1999commandtalk,
  author = {Amanda Stent and John Doweling and Jean Mark Gawron and Elizabeth Owen Bratt and Robert C. Moore},
  title = {{The CommandTalk Spoken Dialogue System}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {1999},
  doi = {10.3115/1034678.1034713},
}
Figure